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DETERMINATION OF OPTIMAL TEST LENGTH TO MAXIMIZE 
THE MULTIPLE CORRELATION 


PAUL HORST 
UNIVERSITY OF WASHINGTON 


If the lengths of the tests in a battery are altered, their inter- 
correlations and their validities or correlations with a criterion are 
also altered. Consequently, the multiple correlation of the battery 
with the criterion will also be altered. These changes are a function 
of the reliabilities of the tests. Suppose we have given from a set 
of experimental data (1) the time allowed for each test in the bat- 
tery, (2) the reliability of each test, (3) the intercorrelations, and 
(4) the validities of all the tests. If we specify the over-all testing 
time we are willing to allow for the test in the future, we can deter- 
mine the amount by which each test must be altered in order to give 
the maximum multiple correlation with the criterion. The method is 
— together with numerical examples and the mathematical 
proof. 


I. The Method 

In general, when we prepare a battery of tests to predict a given 
criterion the resulting multiple correlation is not the highest we could 
get for a given amount of testing time. By altering the lengths of 
the tests we change their reliabilities and consequently also their 
validities and intercorrelations. Alteration of the intercorrelations 
and validities will also change the multiple correlation. Since the 
amount of administration time required of a test battery is often of 
considerable importance, it would be well to know for any particular 
battery whether we could readjust the lengths of the tests without 
increasing the over-all testing time so as to increase the multiple cor- 
relation with the criterion. Suppose we have given the following 
data: 
The time required for each test 
The reliability of each test 
The intercorrelations of each test with every other 
The validity of each test 

5. The total time we are willing to assign to the battery. This 
may or may not be the same as the time assigned to the original bat- 
tery. 
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With these data we can determine how to alter the time for each 
test in order to maximize the multiple correlation of the altered bat- 
tery with the criterion. We assume in this development that if the 
length of a test is altered by a given percentage, the time allotted 
to the test is altered by the same percentage so that for our purposes 
“length of test” and “time for test” are regarded as interchangeable. 
We assume also, of course, that in altering the time for each test we 
do not change the function which it measures. Specifically, we assume 
that in changing the length of a test its correlation with another test 
will be indicated by 





a 
fe | Ti25 
V + (a—1)7,, 





where 


7,2 is the origina] correlation between the two tests, 
ais the new length of the first test divided by its original 
length, 
r,, is the reliability of the first test before its length is altered, 
R,, is the correlation between the two tests aftr the first has 
been altered. 


If both tests have been altered, then the new correlation will be 








ab 
wy ie + (@—1)r,] {1+ —Dral 
where now b is the proportional change in the second test and 72. its 
original reliability. 

Both of these formulas are well known and can be found in any 
good text in psychological or educational statistics. 

First we shall indicate the procedure for determining the altered 
lengths of the tests, including a numerical example. Later the proof 
of the method will be developed. We let 


a; be the proportion of the original total testing time required 
by test 7, 

r;; be the reliability of test 7, 

7; be the correlation of test i with test 7, 

ri. be the validity of test i or its correlation with the criterion , 

T be the ratio of the new total testing time to the old, 

b; be T times the proportion of the new testing time required 
by test 7. 
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From these definitions it should be clear that 


sSa=—1 

So=T. 
Furthermore, we let 

u;=1—7Ti 


and call w; the unreliability of test 7. 

We shall first indicate the procedure for determining the b,’s in 
terms of a solution for three independent variables. The solution can 
be generalized to any number of variables. First we write the equa- 
tions 


a,u V GU, Apu» . V aU, U,0;U; 
(ra +S a+ (ra +~ T Jat (ta + =) a= 


Q,U,0.Us AyU. UM oA: Ub 
ae ee it (tet a ) a + — Ye Us = Noe . (1) 


VG,U,0;Us ) ( V AyU,A;U; a it Als ) 
V3 + mea ra To, + — 2 r: — = Te 
( T Pr 2 T ps se T p= 3 ; 


It will be noted that equations (1) are somewhat similar to a set 
of normal equations used in solving for conventional # regression 
weights. The right-hand sides of the equations are the validity co- 
efficients. The left-hand sides include the intercorrelations. How- 
ever, the diagonal terms on the left contain the reliabilities instead 
of unity. Furthermore, to each coefficient is added a term of the type 


























VAUiAjU; ' : ; ; ; 

— It will be noted that in the diagonal terms, i = 7, and 
VAU;AjU; , AU; 

therefore — is simply 7, 


If we assume now that all the values in equation (1) except the 
6’s are known, we can solve for the f’s by any method desired. After 
solving for the /’s we solve for the b’s by means of the equations 
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b, =f, Vua, — 


> BiVuia; 


— T 
b, = Bo V/U2de St (2) 
bs BiV ua, 

1 


eras. ‘fg 
bz, = Bs WUs@3 —————— 





> Bi Vue; 
1 


The b’s in equations (2) tell us now by what proportion to 
change the time of each test in order that the total testing time be T 
times that of the original, and the multiple correlation of the tests 
with the criterion be a maximum. 

The actual value of the new multiple correlation can be deter- 
mined by the equation 


R,? = ph, Tic + Bs Loe ss Bs Tse + (3) 


Formula (3) is analogous to many published formulas. 
We shall now illustrate the procedure numerically with an ex- 
ample having two independent variables. We let 


a, = 2 a =28 

11, = .60 Yoo = .80 

112 = .20 

?,,-= 40 Yo, = .80. 
Then 

a, = .40 ene, OM 


First we shall take the case of JT = 1; that is, we do not wish 
to change the over-ali time for the test. Our equations (1) would 
then be 


(.60 + .4X .2)f, + (20 + VAX 2X2 XB) Bo = .40 
(.20+ V.4X 2X 2X .8)p. + (.80 + .2 X .8)8=.8 
or 
680 f, + 313 B. =.40 
313 Bb; + .96 Bo = 80). 


Solving these equations for f, and /, we have 
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fp, = 528, B.=.142. 
Using equations (2) to solve for the b’s, 
.523 X V/.08 
= J = .722, 





"523 X V/08 + 142 V6 


= .278. 





7 142 X V.16 
"(523 X 08 + 142 V6 


We see then that the relative administration times of .2 and .8 
for tests 1 and 2, respectively, should be changed to .722 and .278. 
To calculate the ratio of the new times to the old, we have 


o, tae 
——=— > 3.61, 
ay ys 

b, 298 
—=—= .35 
a. 8 


Therefore test 1 would be more than tripled, while test 2 would be 
only about one-third of its original length. 
The square of the multiple correlation for the altered tests would 


be as given by equation 3. 
R,? = .523 X .4 + .142 X .3=.25. 


This compares with .21 for the tests of original length and repre- 
sents an improvement of about 20% . 

Suppose now we are willing to double the testing time so that 
T == 2. Using the same numerical example, we have for equations (1) 




















AX 2 VAX2X2X8 
(.60 + ) a+ (20+ ; ) =.40 
AxXx2X 2x8 2X8 
(20+ Jas (804 )i.=3 
2 2 
or 
640 B, + .256 6. =.40 
256 B, + .880 B. =.30. 


Solving these equations for the {’s, we have 
B; — 552 , Bs» — 181 . 


Using equations (2) to solve for the b’s, 
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. 
i | 
lo 
a 

om) 
CO 
bo 


b,=— Ni aera —— = 1.368, 
552 X \/.08 + .181 X v.16 


181 X v.16 X 2 


b, = —$_____—____—-= 632. 


552 X \/.08 + .181 X \/.16 





We have then 


bh, 368 

— =—— = 6.84, 
(l, A 

b .632 
ae ID, 
a 8 


Therefore, if we double the original testing time, test 1 would be in- 
creased almost seven times while test 2 would be reduced to only 79% 
of its original time. 

If we double the over-all testing time, we have for the square 
of the multiple correlation 


R,? = .552 X 4 + 181 X 3 = .275 


as compared with the original of .21. 

It should be pointed out that the actual computations involved 
in the procedure for determining optimal test length are negligibly 
greater than when regression weights and the multiple correlation 
are calculated from the original data. Furthermore, these computa- 
tions yield also the regression weights and the multiple correlation 
coefficients for the tests of altered length. 


II. Proof of the Method 


We let 
r = the matrix of intercorrelations of the original tests, 
p = the matrix of intercorrelations of the altered tests, 
r, = the vector of validity coefficients of the original tests, 
p- — the vector of validity coefficients of altered tests, 
D, = a diagonal matrix of the lengths of the original tests, 
D, = a diagonal matrix of the lengths of altered tests, 
D= D, D," = a diagonal matrix of the ratios of the new to the 
old lengths, 
D,.. =a diagonal matrix of the test reliabilities. 


We let 
6=[I + (De—1)D,,,] De. (1) 


It can readily be proved then that 
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ped re, (2) 
for the typical element of (2) is 
ei 
ics =i + (@—1)%;; ia - 


which is a well-known formula. 
Similarly, it can also be proved that 


p=d'(r+ d)d+, (4) 


where d is a diagonal matrix which will make the diagonals of p unity, 
that is, 


6*(1 + d)6¢*=I] (5) 
or 
d=6é6—I. (6) 
From (1) and (6) 
d=[(U—D,,) + De(D,,, —I)] De". (7) 
Let 
Ry =i D,,. (8) 


be a diagonal matrix of the unreliabilities of the tests. From (7) and 


(8) 


d=D,D-—D,.. (9) 
Substituting (9) in (4), 
p=d?(r—D, + DID“) 5°, (10) 
or, remembering that D. = D,D, , we have from (10) 
p= 6(r—D, + D,D.D;) 6" . (11) 


We let B be a vector of the beta regression weights for the tests of 
altered length so that 


pB = pe. (12) 
Substituting from (2) and (11) in (12) 
63 (7 — D, + DDD") 6° B = 3... (13) 


It can be proved that the multiple correlation for the tests of altered 
length is given by premultiplying (13) by B’ so that 


B's (r — D, + D, DD") 6*B = B's*r,. = R,? . (14) 


Now let 
B= 68 (15) 
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and substitute in (14): 
B (r—D, + DDD") B= Br. = Ri? . (16) 


Our problem now is to determine the D, matrix in (16) so as to 
maximize R,?. From (16) we get 


B (r— Dy) B + 6 (DuD.Di") B = Re? (17) 
We specify that the sum of the b’s shall be a constant T so that 
T=i1D,\1 , (18) 


where 1 is a vector all of whose elements are unity. 


Letting 4 be the Lagrangian multiplier, we write 


R,? + iT =y, (19) 
or substituting from (17) and (18) in (19) 
p=Pp (r—D,z)b t+ B(DuDDi)B + Al’ Dol. (20) 


Formally we should now differentiate (20) partially with re- 
spect to each of the b’s and equate to zero in order to qbtain values 
for the b’s which will give a maximum for (17) subject to the condi- 
tion (18).* But since the first term on the right-hand side of (20) 
is independent of the b’s, we can consider another function 


o=y—B (r—Di)B 
and write in scalar notation 


Pi? UW, A, =o? Uz Ay 








oD —_— + —————_ + ... + A(b, -+ b, sss), (21) 
b, bs 
Differentiating (21) partially with respect to the b’s, we have 
op —f,2?uU,a, 
— — — -+ i, 
0b, b,? 
(22) 

ow — Po” Uz a 
— = —_—— +4, 
0b. b.? 


ete. 


We have then, equating (22) to zero 


*Osgood, William F. Advanced calculus. New York: McMillan, 1928, p. 180. 
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B, (U, @,)* 
a. 
23) 
Bo (Us Az)? 


2? 





etc. 
Specifying a value } b, we have from (23) 


‘ _ a Bi (Ui ai)? 


epee 24) 
> 5; 





or in matrix notation, where 1 is a vector whose elements are all 
unity, 
; ie, D,)* 8 


jee —, 25) 


YDs1 





Letting Dg be a diagonal matrix of the /’s, we can now write 
1D, 1 








D,= Ds D3) DS —————. (26) 
1'(D, D.)' B 
From (12) and (15) 
(7—D, + D, Da Di") B=. (27) 
From (26) and (27) 
(1'(D, Da)? B) 
r—D,)6+ D,D,.D-' De Dg § ———— “SEH, 28 
“ ” ee Pees - 
But 
Dg f=1. (29) 
Substituting (29) in (28), 
ee (D, Di tl (Du Di)? on 
_— ¥ ) —_———-————— ———————— § —Fr 30 
es VD,1 
or 
CD Ds iE Cp, PD) ) 
r—D +———-— Fan's 31 
( VD, 1 on 
Solving (3) for 6, 
D,D.)*11' (D, Dd.) SY 
1'D,1 
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Equation (31) is the generalized matrix equation corresponding 
to equations (1) of Section I. Equation (32) simply shows the for- 
mal solution for /. 

Equation (26) is the generalized matrix solution for the b’s cor- 
responding to equations (2) of Section I. 

Finally, the matrix solution for R,? is 


i =i (33) 


i 


and corresponds to equation (3) of Section I. 
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NOTE ON THE COMPUTATION OF THE INVERSE OF A 
TRIANGULAR MATRIX 


BENJAMIN FRUCHTER 


AIR FORCE TRAINING COMMAND 
38309TH RESEARCH & DEVELOPMENT GROUP 
UNIVERSITY OF TEXAS SUB-UNIT 


A simplified method of computing the inverse of a triangular 
matrix is presented. It is useful with the multiple-group method of 
factoring the correlation matrix as well as with other factor-analysis 
and multiple-correlation problems. 


The computation of the inverse of a triangular matrix arises in 
several multiple-correlation and factor-analysis problems. One im- 
portant application is the multiple-group method of factoring the cor- 
relation matrix (1). In this method a table of cosines of the angular 
separations of the oblique axes is obtained. Thurstone, in his exam- 
ple of extracting three factors simultaneously, (1, 75), gives the table 
of cosines of the angular separations of the axes (F,,.) as 











P, Po Ps 
Pp, 1000 467 «342 
p, 467 1000 487 =R,. 
p, 842 487 1.000 





It is then desired to find the matrix which will transform the 
oblique factors to an orthogonal frame of reference. This is accom- 
plished by factoring matrix R,, by the diagonal method. The result- 
ing matrix is (Fyn): 





I I =o 





Pp, 1000 0 aaa 
p, 467 884 0 =F... 
Ps 342 814.886 





The inverse of the transpose of matrix F',,, is the desired trans- 
formation matrix. 

Several methods have been proposed for calculating the inverse 
of a matrix (2, 3). They are laborious and limit the convenience of 
the multiple-group method when a large number of factors are to be 
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extracted simultaneously. For a triangular matrix, however, the pro- 
cess can be simplified and the computation of the inverse of relatively 
large matrices is not prohibitive. 


Computational Procedure 
By way of illustration, the inverse of matrix Fp», above will be 
computed. 
The inverse of the transpose of this matrix may be found directly 
from F'y», as follows: 
Represent matrix F’,,, by matrix A: 











I II III 
Pi 1.0* 0 0 
P, A, A509 0 
P G34 G35 O35 





*The value in cell @: is al- 
ways 1.0 for this type of 
problem. 


Represent the inverse of this matrix (F'pm)-? by A+ 








I II III 





Py 34 %19 %13 
P, %o4 %o0 Xo 
Ps 34 Vso X33 





Then A - A —I, or, written out: 














= Ss 2 l wom I Il Il 

 p, 10 0.0 0.0 Si, Vie Bag 10 0.0 0.0 (1) 
Po %, Myo 0.0 ° 9 ee sae EunD 
ee Vas. Cae. war 0.0 0.0 10. 





Performing the row by column matrix multiplications gives the 
following equations: 
(Row 1 X column 1) 


1.0 11 a 0.0 Vo1 ar 0.0 31 = 1.0 (2) 
1.0 
%4,=—=1.0. 3) 
1.0 ¢ 


(Row 2 X column 1) 


ny + Aqo%2, + 0.0 3, = 0. (4) 


As was shown in equation (3), 7,,—1.0. Hence 
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ei + Age%o1 — 0 
1 AGT 
tp =— = = 529 
dx, «884 


(Row 3 X column 1) 
Qgi%y1 1 AgeXe1 + Ags%s, = 0. 


From equation (3) #11 = 1.0, and from equation (6) 





Agi 
Xo an ee 
Qe 
Substituting and transposing 
, G32Me1 
O33%3) = — Ag, T 
Qz2 
and 
a a 
Xs1 SE ence ue a Golo, —— 
G3, Aan Ass 


342 314 X .467 


— = — .3886 + .187 = — .199. 


886.884 X 886 
(Row 1 X column 2) 
X12 = 0.0. 
(Row 2 X column 2) 
2X12 +Oe2%oo + 0.0 oe = 1.0. 
From equation (10) x,. = 0; hence 
1 1.0 
Lee gr ae 1.131. 
(Row 3 X column 2) 
AgiL32 + AzeXoo + Az3%z2— 0. 


From equation (10) x:. = 0, and from equation (12) 


Substituting 


91 
(5) 
(6) 


(7) 


(8) 


(9) 


(10) 


(11) 


(12) 


(13) 
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Az9 
— + Q33%32 = 0, 





Are 
Az. 014 
¢32=— =— =—.401. 
Az2M33 .884 X .886 

(Row 1 X column 3) 

X13;— 0.0 
(Row 2 X column 3) 

Les = 0.0 


(Row 3 X column 3) 


31X13 + Azo%e3 + Asz3%33 = 1.0. 


From equations (16) and (17), 4:3; = 0 and x; =0. 


Therefore 


1 1 
£33 > OO 1,229. 
a; .886 


Putting the results in the form of a matrix gives 











I II III 
P, 1.000 000 = .000 
Pp, —.529 1.131 000 = (Fy)-) « 
P, —199 —401 1.129 





(14) 


(15) 


(16) 


(17) 


(18) 


(19) 


This is the inverse of Fp». Comparing matrices Fm and (F'pm)7, 
it may be observed that wherever a zero occurs above the principal 
diagonal in the former it also occurs in the latter. The values along 
the principal diagonal of (Fp)? are the reciprocals of the corre- 
sponding values of Fyn. The other values of (Fyn)! are obtained by 


means of simple equations similar to those outlined above. 


The desired inverse is merely its transpose (F"’,,)-! and is writ- 














ten 
I II III 
DP, 1.000 —.529 —.199 
ee? ORR oe EL. 


p, 000 000 1.129 
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A METHOD OF MATRIX ANALYSIS OF GROUP STRUCTURE 


R. DUNCAN LUCE AND ALBERT D. PERRY 


GRADUATE STUDENTS, DEPARTMENT OF MATHEMATICS 
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 


Matrix methods may be applied to the analysis of experimental 
data concerning group structure when these data indicate relation- 
ships which can be depicted by line diagrams such as sociograms. 
One may introduce two concepts, n-chain and clique, which have sim- 
ple relationships to the powers of certain matrices. Using them it 
is possible to determine the group structure by methods which are 
beth faster and more certain than less systematic methods. This 
paper describes such a matrix method and applies it to the analysis 
of practical examples. At several points some unsolved problems in 
this field are indicated. 


1. Introduction 

In a number of branches of the social sciences one encounters 
problems of the analysis of relationships between the elements of a 
group. Frequently the results of these investigations may be pre- 
sented in diagrammatic form as sociograms, organization charts, 
flow charts, and the like. When the data to be analyzed are such that 
a diagram of this type may be drawn, the analysis and presentation 
of the results may be greatly expedited by using matrix algebra. This 
paper presents some of the results of an investigation of this appli- 
cation of matrices. Initial trials in the determination of group struc- 
tures indicate that the matrix method is not only faster but also less 
prone to error than manual investigation.* 

The second section of this paper presents certain concepts used 
in the analysis and associates matrices with the group in question. 
The third states the results obtained and the fourth gives illustra- 
tions of their application. Finally, section five contains a mathemati- 
cal formulation of the theory and derivation of the results presented 


in section three. 


2. Definitions 
2.01. The types of relationships which this method will handle 
are: man a chooses man b as a friend, man a commands man b, a sends 
messages to b, and so forth. Since in a given problem we concern 
*Some of these examples have been worked out by the Research Center for 


Group Dynamics, Massachusetts Institute of Technology, in conjunction with 
some of its research. 
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ourselves with one sort of relation, no confusion arises from replac- 
ing the description of the relationship by a symbol =>. Thus, in- 
stead of “man i chooses man j as a friend,” we write “i = > 7.” If, on 
the other hand, man i had not chosen man j, we would have written 
“¢ # > 4,” using the symbol # > to indicate the negation of the rela- 
tionship denoted by = >. 

2.02. Situations such as mutual choice of friends or two-way com- 
munication would thus be indicated by i= > 7 and 7 = > i, or briefly, 
4< => j. We describe such situations by saying that a symmetry 
exists between 7 and j. 

2.03. When the choice is not mutual, thatisi=>jorj=>i 
but not both, we say an antimetry exists between i and 7. 

2.04.* The data to be analyzed are presented in a matrix X as 
follows: the i,j entry (x;;) has the value of 1 if i= > 7 and the value 
0 if i+ > 7. For convenience we place the main diagonal terms equal 
to zero, i.e., x;; = 0 for all 7. This convention, 1 # > 7, does not re- 
strict the applicability of the method, since there is little significance 
in such statements as “Jones chooses himself as a friend.” 

Suppose, for example, that we had a group of four members with 
the following relationships: a= > b,b=>a,b=>d,d=>b, 
c=>a,c=>b,d=>a, and d= >. All other possible combina- 
tions of a,b,c, and d are related by the symbol # >. The X matrix 
associated with this group is: 


a > ce a 
a 0.2.0 @ 
b 1 @ @ 2A 
c = ae ee 
d i 2 t.® 


2.05. From the X matrix we extract a symmetric matrix S hav- 
ing entries s;; determined by $;; = s;; = 1 if x:; = x;;=1, and other- 
wise $;; = s;; = 0. All the symmetries in the group are indicated in 
the matrix S. The S matrix associated with the above X matrix is: 


0100 
ieea! 
0000 

eee 


*In the course of the present work it was brought to our attention that in 
“A matrix approach tc the analysis of sociometric data,’ Sociometry, 1946, 9, 
340-347, Elaine Forsyth and Leo Katz have used matrices to represent socio- 
metric relations. They considered a three-valued logic rather than the present 
two-valued one, and the operations on the matrices are different from the ones 
discussed in this paper. 
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To indicate the 7,7 entry of the matrix X", which is the n* 
power of X , we shall employ the symbol x;;. Similarly, the 7,7 en- 
try of S* is 3;;™. 

2.06. In the group considered above, we hada =>b,b=>d, 
and d = > c as three of the relations. If the symbol = > indicates the 
relationship “sends messages to,” it appears that a can send a mes- 
sage to c in three steps, via b and d. We call this three-step path a 
3-chain from a to c. Rather than writing out the above sequence of 
relations, we may omit the symbol = > and simply write the 3-chain 
as a,b,d,c . 

In a group involving more elements one might have the 5-chains: 
a,e€,c,b,d,f and a,d,b,c,d,e. We notice that the first sequence involves 
five steps between six elements of the group. The second sequence 
also involves five steps but only five elements of the group, since the 
element d appears as both the second and fifth member of the se- 
quence. Thus, although these two five-step sequences contain differ- 
ent numbers of elements of the group, they both have six members. 
Using this concept of membership in a sequence, an n-step sequence 
has n+1 members. 

These examples of 3-chains and 5-chains suggest a general defi- 
nition for a property within the group: an ordered sequence with n+1 
members,7,a@,6,-++,p,q, 9, is an n-chain from i to j if and only if 


‘=>4,4=>0D,---,p=>q¢,q=>>). 


2.07. When two n-chains have the same elements in the same 
order, i.e., the same members, then they are said to be equal, and 
otherwise they are distinct. It is important in this definition of equal- 
ity that it be recognized that both the elements of the group and their 
order in the sequence are considered. The two chains 2,7,k,l,o and 
i,p,k,j,l are distinct though they contain the same five elements. 

2.08. When the same element occurs more than once in an 
n-chain, the n-chain is said to be redundant. (Thus, in a group of m 
elements any n-chain with n/greater)than m is redundant). The chains 
a,b,e,d,b,c and 4@,c,a,b,d,c,e are, for example, both redundant, for the 
element 6b occurs twice in the former and the elements a and ¢ both 
occur twice in the latter. An example of a non-redundant 5-chain is 
a,d,p,b,q,9 . 

2.09. A subset of the group forms a clique provided that it con- 
sists of three or more members-each in the symmetric relation to 
each other member of the subset, and provided further that there 
can be found no element outside the subset that is in the symmetric 
relation to each of the elements of the subset. The application of this 
definition to the concept of friendship is immediate: it states that a 


\ A 
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set of more than two people form a clique if they are all mutual 
friends of one another. In addition, the definition specifies that sub- 
sets of cliques are not cliques, so that in a clique of five friends we 
shall not say that any three form a clique. Although the word “clique” 
immediately suggests friendship, the definition is useful in the study 
of other relationships. 

2.10. This definition of clique has two possible weaknesses: 
first, if each element of the group is related by = > to no more than 
c other elements of the group, then we can detect only cliques with 
at most c + 1 members; and second, there may exist within the group 
certain tightly knit subgroups which by the omission of a few sym- 
metries fail to satisfy the definition of a clique but which nonetheless 
would be termed, non-technically, “cliques.” It may be possible to 
alleviate these difficulties by the introduction of so called “n-cliques” 
which comprise the set of n elements which form two distinct n-chains 
from each element of the set to itself. This requires that the n-chains 
be redundant with the only recurring element being the end-point 
and also that all the relations in the n-chains be symmetric. 

This definition means that the four elements a,b,c, and d form 
a 4-clique if the 4-chains (for example) a,b,c,d,a and a,d,c,b,a, both 
exist. These by the definition of n-chain require that the relations 


e<=>b,6<=>e,¢< =>d,€<—>—>6 


exist, but nothing is said about the relations between a and c, and 
b andd. The original definition requires, in addition, that 


as=>ce end 8 > >< 


for a,b,c, and d to form a clique of four members. Thus we see that 
the definition of n-clique considers “circles” of symmetries, but it fails 
to consider the symmetric “cross” terms that exist between the mem- 
bers of the n-clique. These cross terms will be investigated, however, 
by determining whether any m of these n-elements form an m-clique. 

The usefulness of the definition of n-clique can be judged only 
after experience has been gained in its application. This is not con- 
veniently possible at present, unfortunately, because the problem of 
the general determination of redundant n-chains has not been solved 
(see §5.09). 

The most general definition of a clique-like structure including 
antimetries will not be discussed, for it is believed that this will not 
be amenable to a concise mathematical formulation. 


3. Statement of Results 
3.01. In X” the entry «;;°" = ¢ if and only if there are ¢ dis- 
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tinct n-chains from i to 7 (for proof see $5.04). Thus, if in the fifth 
power of a matrix of data X we find that the number 9 occurs in the 
third row of the seventh column, we may conclude that there are 9 
distinct 5-chains from element 3 to element 7. 


a 3.02. In X? the i" main diagonal entry has the value m if and 
only if 7 is in the symmetric relation with m elements of the group 
(§5.05). Since by the definition of a clique each element 7 in a 
clique of t members must be in the symmetric relation to each of the 
t—1 other elements, it is necessary that x;;°) > t—1 for i to be ina 
clique of ¢ members. We may not, however, conclude from the fact that 
x;;° > t—1 that 7 is necessarily contained in a clique of ¢ members. 


3.03. An element 7 is contained in a clique if and only if the 7” 
entry of the main diagonal of S* is positive (§5.06). The main diag- 
onal terms of S* will be either 0 or even positive numbers in all cases, 
and when the value of the entry is 0 the associated element is not in 
a clique. 


3.04. If, in S*, ¢ entries of the main diagonal have the value 
(t—2) (t—1) and all other entries of the main diagonal are zero, then 
these t elements form a clique of t members (§5.08). It also follows 
from the next statement (§3.05) that if there is only one clique 
of ¢ members then these ¢t elements will have a main diagonal value 
in S*° of (t—2) (t—1). The former statement is, however, the more 
significant in analysis, for it is the aim to go from the matrix repre- 
sentation to the group structure. There is no difficulty in going from 
the structure to the matrices. 

3.05. Since by statement 3.03 the main diagonal values of S®* 
are dependent only on the clique structure of the group, it is to be 
expected that a formula relating these values and the clique struc- 
ture is possible. If an element 7 is contained in m different cliques 
each having t, members, and if there are d, elements common to the 
k‘* clique and all the preceding ones, then 


sii = 3 { (ty — 2) (tr —1) — (dy —2) (dy —1)} +2 


($5.07). Thus, if we have three cliques: (5,7,9,10), (1,4,9), and 
(1,2,5,9,11), then d, = 0, for there are no preceding cliques; d. = 1, 
for only element 9 is common to the second and first cliques; and 
d, = 3, for clique three has the elements 1,5, and 9 common with 
the first two cliques. Substituting ¢, = 4,4—3,t,=—=5,daq=—0, 
d, = 1, and d, = 3 and evaluating the formula for element 9, which 
is the only one common to all three cliques, we obtain 
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)¢ 
+ [(8—2)(8—1) — (1—2)(1—1)] 
+ [(5—2)(5—1) — (8—2)(3—1) ] +2 
= 28. 


In the evaluation of this formula it is immaterial how the cliques are 
numbered initially: however, it is essential once the numbering is 
chosen that we be consistent. 

3.06. The redundant 2-chains of a matrix X are the main diago- 
nal entries of X* ($5.09). Thus for a matrix 





2 8.2% 
ee a a | 
x= 1000 1 
76 6 ¢ 1 
| ;i 8 9 3 
with the square 
[ 2 @ 42-2 2 | 
| Ss @ Dh. 2 
x=!120101, 
. tt @- 2,8 | 
2A &:O Og 


the matrix of redundant 2-chains is 
i © @® 8 ® 
2 ® ®@.-@ 
o 0 0 6 @ 
ae eee | 
e880. @.2 
To obtain the matrix of redundant 3-chains we compute the fol- 
lowing matrix, in which the symbol R® stands for the matrix of re- 


dundant 2-chains: 
XR® +ROX—S. 


Deleting in this sum the main diagonal and replacing it by the main 
diagonal of X* gives the matrix of redundant 3-chains ($5.09). If 
the main diagonal of XR” + R®X — S is denoted by Y and the 
main diagonal of X“ by Z®, then let EF = Z°) — Y® and thus 


> 


the matrix of redundant 3-chains, R“, is given by 
R® - XR” \- ROX t E® —S 7 
It has not yet been possible to develop formulas which will give 


the matrix of redundant n-chains for n larger than 3. What work 
that has been done in this direction is presented in §5.09. 
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3.07. The several theorems on cliques give a method that to some 
extent determines the clique structure independent of the rest of the 
group structure. It would be desirable to find a simple scheme that 
determines the clique structure directly. Since a certain amount of 
knowledge in this direction can be obtained from S*, it is conjec- 
tured that possibly there is a simple formula relating clique struc- 
ture to the numbers in S*. As yet no such formula has been developed. 

In a consideration of this problem, it was questioned whether 
certain aspects of the structure would be lost in the multiplication, 
which, if true, might make the discovery of the desired formula im- 
possible. The following theorem shows that neither the clique struc- 
ture nor any of the properties of S are lost in the matrix S*: Any 
real symmetric matrix has one and only one real symmetric 7 root 
if n is a positive odd integer (§5.12). This theorem is somewhat more 
general than was required, since it does not restrict the entries in 
the n root to 0 and 1, and since it is true for any odd root rather 
than just the cube root. (In general the real symmetric even roots 
are not unique.) 

This theorem suggests a further problem to be solved: to find 
a symmetric group structure which will insure the presence of cer- 
tain prescribed minimum n-chain conditions for odd . To carry this 
out it will probably prove necessary to discover a theorem that uses 
not only the realness and symmetry of the S matrix and its powers, 
but in addition the fact that only the numbers 1 and 0 may be entries 
nS. 


4. Examples 
4.01. As the first example, let us compare and analyze the friend- 
ship structure in the two following hypothetical groups. The matrices 
are (where a blank entry indicates a zero): 
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The associated S matrices are: 
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Here the differences between the groups are becoming evident. 
In group I, men 3 and 9 have no mutual friends, since 8332) = 859°?) = 0 
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(§3.02). Thus, as far as symmetric relationships are concerned, these 
men are isolated from the group. In the same way we determine that 
5 and 6 each have just one symmetric friendship relation (s;;‘ = 8.6? 
= 1, §3.02) which we determine to be 5 <=> 6 from the S matrix. 
The remaining elements in S? form a rather dense set of quite large 
numbers, which means, roughly, a tightly knit group. 

In the second group, on the other hand, every man has a non-zero 
main diagonal in S*. The men 2, 5, 7, 8, and 10 each have a single 
mutual friend, which we determine to be: 2 <=>6,5 <=> 1, 
7<=>6,8<=>9,and10 <=> 1. Then since s,,°?) = 2 and since 
we have just cited 6’s two mutual friends, 6 need not be considered 
further. We note that the off-diagonal areas of this S? matrix are 
not so completely filled as group I, indicating that the group is not 
so tightly bound. 

The S* matrices indicate the differences in compactness of the 
structures quite clearly: 


I Il 

izsase te oo 12345678910 
aE oe Ww et wy 1 2 564 iis 
2/1514 14 1412 12 2 2 
3 gi 6 241 ae 
4] 1414 10 os +t 4/6 421 41 
5 1 ss" 3 1 
6 1 6 2 2 
7/1414 18 1010 8 7 2 
:s,net ¢ we Ff th ae: 2 
9 fir +a ¢°% 
10 | 1212 10 8 7 6| 10] 4 1 1 











Since the corresponding main diagonal terms are.non-zero, men 1, 2, 
4, 7, 8, and 10 of group I are in cliques (§3.03). These, with 3 and 9 
which have no symmetries in the group and 5 and 6 which are mutual 
friends, account for all members of the group. The terms 8x3°°) = 
Sion = 6 suggest a clique of four members; however, the existence 
of other main diagonal terms makes it impossible to apply the for- 
mula (t—2) (t—1) (§3.04). Investigating in S first the elements 1, 
2, and 4 because their columns have the largest values in the tenth 
row, we find that elements 1, 2, 4, and 10 form a clique of four mem- 
bers. In the eighth row the largest entries are in columns 1, 2, and 
7, and an investigation reveals that 1, 2, 7, and 8 form a clique of 
four men, which then overlaps the first clique by the men 1 and 2. In 
row four the largest entries are found in columns 1, 2, and 7. We 
then find that 1, 2, 4, and 7 form a clique of four elements which 





; 
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overlaps the previous two. All the men contained in cliques have been 
accounted for at least once, and a check either with the formula for 
main diagonal entries ($3.05) or directly in the S matrix indicates 
that all the cliques have been discovered. This, coupled with what 
we discovered in S*, completely determines the symmetric structure 
of the first group. 

For purposes of qualitative judgment and a guide to carrying 
out analysis, we note that the first two rows of S® present an inter- 
esting summary of the clique structure. The entries s,.“ and 82,“ 
have the largest values, next largest are in columns four and seven, 
and then finally in columns eight and ten. Men 1 and 2 are contained 
in all three cliques, 4 and 7 are each contained in two cliques, and 
finally men 8 and 10 are each in only one clique. This indicates that 
the magnitude of the off-diagonal terms determines to some extent 
the amount and structural position of the overlap of cliques. 

In group II there are only three elements with non-zero main- 
diagonal entries, all with the value 2. This fits the formula (t—2) 
(t—1) with ¢ = 3 (§3.04). Thus the men 1, 3, and 4 form a clique 
of three members. Returning to S?, we see that there remains one 
unaccounted symmetry each for men 4 and 9, hence 4 <<=>9. 

In group I, the off-diagonal terms are large in magnitude and 
are quite dense in the array, with some rows completely empty or 
with single entries in the S* matrix. This indicates a closely knit 
group with certain men definitely excluded. The S* matrix for the 
second group has fewer entries of a smaller value indicating a less 
tightly knit structure, but it has no empty rows and only one row 
with a single entry; that is, it has fewer people than group I who 
are not accepted by the group or who do not accept it. 

A consideration of the matrix X — S will give all the antimetries 
in the groups and complete the analysis of the structures. 

It is clear that-this procedure gains strength as the complexity 
of the preblem increases, for the analysis of a twenty-element group 
is little more difficult than that of a ten-element group. 

4.02. The second example is a communication system compris- 
ing two-way links between seven stations such as might occur in a 
telephone or telegraph circuit. The number of channels of a given 
number of steps (i.e., n-chains in the general theory) between any 
two points and the minimum number of steps required to complete 
contact between two stations will be determined. Suppose the matrix 
of one-step contacts is: 
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1234567 
a 11 a 
si % 3 
ei dE. 22 
4 1 111 
5 ‘2 3% 
6 te 1 
rte: eek 








which in this case is also the S matrix. Then two-step connections 
are given by X?: 








12346567 
a ee a 
2 ce See F 
3 43241123 
4 31242322 
5 st ise23 
6 toe2232 
71, 01382224 | 

and the three-step ones by X?: 

2 ts @ §¢ 7 
1 se 8 € €4 8") 
2 43638 $8 38 
3 85 410105 5 
4 4310 8 9911 
5 4310 9 8911 
6 43 59 96 8 
TL et Swarms © | 








(2) (2) 
From the former, the two connections 1 <=> 7 and 2 <=> 6 can- 
not be realized because 217°) = 271°) = 0 and 2°? = Xo = 0 
(§3.01). The contacts are possible in three steps, however, since X* 
is completely filled. Thus two steps are sufficient for most contacts 
and three steps for all. . 

In determining the number of paths between two points it is de- 
sirable to eliminate redundant paths. For two-step communication 
this is done by deleting the main diagonal of X*. The remaining terms 
represent the number of two-step paths between the stations indi- 
cated. The matrix of redundancies for three-step communication is 
given by R® = X¥R® + ROX + E® — § (§3.06), which works out 


to be: 
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1234667 
17246 67 
2 et 
3165477 
4 78767 
5 7T7O89 
6 6656 ™ . 
ris 7766 | 








The matrix of non-redundant three-step communication paths is 
X3 ae R®) . 


1234567 
1 24442 
2 3333 
3 2 33855 
t 133 234 
D 4332 34 
6 4353383 2 
7 235442 


We notice that the three-step paths between 1 and 2 and 2 and 3 are 
all redundant but that there are two-step paths for these combina- 
tions. All other combinations have at least two three-step paths join- 
ing them. 

5. Mathematical Theory 

5.01 To carry out the following mathematical formulation and 
the proofs of theorems it is convenient to use some of the symbolism 
and nomenclature of point set theory. As there is some diversity in 
the literature, the symbols used are: 

Sets are either defined by enumeration or by properties of the 
elements of the set in the form: symbol for the set [symbols used for 
elements of the set | defining properties of these elements]. When 
a single element 7 is treated as a set it will be denoted by (7), other- 
wise sets will be denoted by upper case Greek letters. 

The intersection of (elements common to) two sets J’ and ® 


is denoted by I’: @,. 

The union of two sets I and ® (elements contained in either or 
both) is denoted by I’ + #. The context will make it clear whether 
the symbol + refers to addition, matrix addition, or union. 

The inclusion of a set J’ in another set ® (all elements of J’ are 
elements of ®) is denoted by 1’ < ®. The negation is I' <* ®, 

If @ < 2, then the complement of © with respect to 2, ®’, is de- 














R. DUNCAN LUCE AND ALBERT D. PERRY 107 


fined by ® + 6’ = 2 and © - & = 0 where 0 is the null set. 
The inclusion of a single element 7 in a set @ is denoted byie®. 
For any two elements 7 and 7 of a set = and a subset 2 of 2: 
(i) + (7) < Qif and only ifie QandjeQQ. 
(i) + (7) <* Q implies i e Q’ and/or 7 ¢ 2. 


The symbol 6;; =1 if t=3j 
=0 if t#7 
5.02. Consider a finite set Z of x elements denoted by 1, 2,-:-, 
i,+++,9,°++, # for which there is defined a relationship = > between 
elements and its negation + > having the properties: 


1. EHithert=>jori#>jforalliandjeZ. 


2. i# >i. 

Let a number 2;; be associated with i and 7 such that 
ad. if i=>j as 

“ =0 if i#>j. 


A matrix X = [ai;] is formed from the numbers x;;. It will be found 
useful to denote the i,j entry of the n'" power of X , X”, by xi;"". 

A symmetry is said to exist between i and 7 if and only if i= > 7 
and 7 = > 7, in which case we may write i < = > 7. For the matrix 
X this requires that x;; = x;; = 1. If, however, either i => j and 
j#>iori# > andj = >i then an antimetry is said to exist be- 
tween 7 and 7. 

The symmetric matrix S associated with the matrix X is defined 
by S = [si;] , where 
o., = 8: ==] if Mig =arji=—1, i.e., i<=>)}. 

“to "“ =0 — otherwise. 

The 7,7 entry of the n power of S is s;;‘"". 

5.03. Definitions: 

1. An ordered sequence with n+1 members, i = 91, y2,-°-+ 5 Yn 
Yui = J, is an n-chain I from i to 7 if and only if 

1=y,>=> Vos Yo= > Yae%**s Y= > Yan = J- 
(n) 
In brief, i => j indicates that there exists an n-chain from i to), 
which may also be enumerated as i = y., 2, ++, Yn» Yau = J, OF, 
when no ambiguity will arise, asi, k,!,---,p,q, 7 with the order- 
ing being indicated by the written order of the sequence. 
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2. Two n-chains I and © are equal if and only if the r* mem- 
ber of I equals the 7* member of ©, i.e., y,=¢,,forl<r<nt+l1. 


If this is not true, then I’ and @ are distinct. 


3. Each pair of elements 7, and y, of an n-chain with 1 <k< 
m<n-+1 and », = ym is said to be the redundant pair (k,m). An 
m-chain is redundant if and only if it contains at least one redundant 
pair. 

4. The elements 1,2,---,¢ (¢ > 3) forma clique 0 of t mem- 
bers if and only if each element of 9 is symmetric with each other ele- 
ment of 9 , and there is no element not in 0 symmetric with all ele- 
ments of 0. 


This is equivalent to 


4ij =1—6;; fori, 7=1,2,---,¢ but not for?,7=—1,2,---,t, 
t + 1, whatever the (¢ + 1)** element. 


5.04. Theorem 1: 2;;°". = ¢ if and only if there exist c distinct 
n-chains from i toj. 
Proof: By definition of matrix multiplication 
. wij =D +++ ZL LinLea +++ Xpghaj, 
kes qe 
with the summations over n—1 indices. Suppose that the indices have 
been selected such thati,k,/,---,»,q,jis an mchain from i toj7. 


Then by definition 1 (§5.03) 
i= Ly SS Ly = Ly 1 . 


and if the indices were not so selected then at least one x,, = 0. Thus 
n-chains contribute 1 to the sum and other ordered sequences con- 
tribute 0. Since the indices take on each possible combination of 
values just once, every distinct n-chain is represented just once. If 
there are c such n-chains, then there are a total of c ones in the sum- 
mation. 


5.05. Theorem 2: An element of Z has a main diagonal value 
of c in X? if and only if it is symmetric with c elements of 2. 
Proof: Let © be the set of 7’s for which i <=> 7. By definition 

xii = > LijX ji + > Lij%,= Di + > 
je? jee’ 

>: = c by theorem 1 ($5.04) and &, = 0 because i and 7 are not sym- 
metric for j « &’, so either x;; = 0 or x;; = 0 or both. Thus if i is sym- 
metric with c elements of 2, 2;; =c. 
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If xii° =c, then by theorem 1 there exist ¢ distinct 7’s such that 
Mii H=eti—1,ie,i<=—> 7 forcy’s. 

5.06. Theorem 3: An element 7 is contained in a clique if and 
only if the i entry of the main diagonal of S* is positive. 
Proof: Suppose that 7 is contained in a clique 0. 
By definition 

859 = TD SijSjnSei- 
()+(k)< 

Select 7 and k& such that (7) + (k) < O and such thati# j7+#k 
#7. Such elements exist by the definition of a clique (definition 4, 
§5.03). It is true by the definition of a clique and of the matrix S 
that: 8;; = 8); = Sj, = 8; = Six = S&; = 1 for such 7 and &k. Thus this 
choice of j and k contributes 2 to the summation, and because s;; > 0 
for all 7 and 7 there are no negative contributions to the sum; there- 
fore si; >2>0. 


Suppose that s;;°’ > 0. Then there exists at least one pair of 
elements of 7 and k such that s;; = s;, = s,; = 1 and this implies 
i<=>=>j,j<=>k, and k <=> i. If there are no other ele- 
ments symmetric with 7, 7, and & then these three form a clique. If 
there is another element symmetric with these three, then consider 
the set of four formed by adding it to the previous three. If there is 
no other element symmetric with these four, they form a clique. If 
there is, add it to the set and continue the process. Since the set = 
contains only a finite number of elements, the process must terminate 
giving a clique containing 7. 


5.07. Theorem 4: If 1) Oo are cliques of ts members, 2) the sets 
Ay = 0, - (09, + O, + ++» + Oy.) have dy members, and 3) 7 is con- 
tained in the cliques 6, ,0—1,2,---, m, then 

85 =F ( (te — 2) (to —1) — (do — 2) (do —1)} + 2. 
Proof: By definition 
85 = J DS $ijSjxSxi. 
(j)+(k) <= 
The set of all the pairs j, k is the union of the following three mu- 
tually exclusive sets: 


Y, [j,k | there exists » such that (7) + (k) < 0,; there does not 
exist a such that (j) + (k) < Oa, (7) + (k) <* Aa] 
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WY. [7,k | there does not exist a such that (7) + (k) < Oc] 

Y. [7, k | there exists a such that (7) + (k) < Oc, (7) + (kK) <* Aa]. 
1. For ¥, then either 


a) (7) + (k) <* ©. for all a. This is not possible because 
b) (7) + (k) < Aa for all a. This is not possible because 
== s 


orc) (j) + (k) < ©. if and only if (7) + (k) < Ac for alla. 
This is not possible because 4, = 0. Thus ¥, is empty. 

2. (7) + (k) < ¥®. implies 8;;8)8%; = 0 for 38ij;8),.8,; = 1 im- 
plies that 7, 7, and & are either a clique or a subset of a clique (by 
the argument of theorem 3), but (7) + (k) < W, implies 7 and k 
are not contained in any clique. 

3. YW, gives that 


> ; ." Y > > 
$ij = SD Si j8jx8ki 


(j)+(k) <¥3 








i 
= =, > > S558 jx8ki 
Vv=1 (j)+(k)< <_F : 


bw j)+(k)<« ohy 





We observe that: 2,[7,k| (/) mi (k) <6.) =2.[7,k | (7) + (kh) < 
ee As k | (7) + (k) < Oy, (7) + (Kk) <* Av] 


and since 2, - 2, = 0, it follows that 5 = 5 + 5 or > == — S- 
Q4 Q2 3 OQ: 1 Q2 

Q, is the set of all ordered pairs (7) + (k) <O,. Ift AZ FARA, 

then si; = Sj, = si = 1, otherwise one of the s,, = 0. Since every 


QO, contains t, elements’, there are 1 -1P2 ordered pairs satisfying these 
conditions. Thus: 
= +,.P2= (ty —2) (t’—1) . 
1 
Similarly 
_ |(dy—2)(d,—1), »>1 
0 y=1 since 4,=—0. 


PM 





Combining these, 


>= (t, —2) (t,—1) — (d,--2)(dy—1),»>1 
Q3 (ty —2) (t.—1) »y—1. 
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Summing over » gives 
sis =3 ( (tr —2) (t—1) — (dy —2) (dy —1)} 
+ (t,—2) (t,—1) 
oa E(t —2) (t, —1) — (d, —2) (dy —1)} + 2. 


Since the entries s;;) are uniquely determined from the entries 
of S by the laws of matrix multiplication, all valid methods of cal- 
culating s;;° will give the same result. Specifically, in the above 
formula the numbering of the cliques is immaterial. 


Similar formulas to that just deduced may be given for the off- 
diagonal terms of S*, but they are considerably more complex, and, 
to date, they have not been found useful in applications. 


5.08. Theorem 5: If 1) O is a set of t members with ¢ > 3, 
2) sii = (2) (1) for 7 contained in ©, and 3) s;;° = 0 for 7 
contained in 0’, then © is a clique of t members. 
Proof: There are two cases: 


1. «<=> /j for alli, je 0, then 9 is a clique by definition 4 
(§5.03) and theorem 3 (§5.06), and it has t members by part 1 of 
the hypothesis. 


2. There exist p and q « 9 such that p and q are not symmetric. 
Then by definition 


SiO = TT SijSjSei 


(j)+(k)<@ 


+ SS $5 jSjxS8ki- 


(j)+(k)<*8 
If $;;3,8,; = 1, the elements 7, 7, and k are a clique or a subset of 
a clique and thus by hypothesis (3) and theorem 3 (§5.06) they are 
all contained in 0; therefore the second sum = 0. Introduce in £ 
sufficient relationships p => q to make 9 a clique & of ¢ members. 
Since s;; > 0 for all i and 7, the introduction of these s,, = 1 must 
increase the sum by 2 or more, for at least two additional 3-chains 


are introduced (i,p,q,7 and i,q,p,1) ; hence by theorem 4 (§5.07) 
Si, os = = 8558 jxSki —2=> (t—2) (t—1) —2 


(j)+(k)<® 


< (t—2) (é-1), 
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which is contrary to hypothesis (2). Therefore © is a clique of t 
members. 


5.09. Redundancies: 


By definition 3 (§5.03) an n-chain is redundant if and only if it 
contains at least one redundant pair (k,m), where a redundant pair 
defines two members of the n-chain »; and ym with 7, = ymandk<m. 
If these ordered subscript pairs (k,m) and the end point pair (2,7) 
(the latter not necessarily a redundant pair) are considered as sets, 
then five classes of mutually exclusive redundant 7-chains may be de- 
fined which include all redundant n-chains: 


1. The A, class: There exists at least one redundant pair (k,m) 
and it has the property: 
(k,m) + (4,7) =0. 
2. The B, class: There exists one and only one redundant pair 
(k,m) and it has the property: 
(k,m) - (4,7) =2. 
3. “The C, class: There exists one and only one redundant pair 
(k,m) and it has the property: 
(k,m) - (1,7) =3. 
4, The D,, class: There exist two and only two redundant pairs 
(k,m) and (p,q) and they have the properties: 
(k,m) - (4,7) =74 
(p,q) - (4,9) =). 
5. The E, class: There exists one and only one redundant pair 
(i,m) and it has the property: 


(k,m) - (4,7) = (47). 


If there are ¢ -chains i ="> 7 of the class A, from i to 7, then 
define a;;" = t. From these numbers the matrix A™ = [a@;;™] is 
formed. This is the matrix of redundant v-chains of the class A,. 
If R™ is the matrix of redundant n-chains it follows, if analogous 
definitions are made for matrices of the other four classes, that 


RO=AM + BOL OM 4+ DM FEM, 


It follows directly from the definitions and the limitations on 
that 
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R® =0 
R® = [65,0452] =i (2) 
A®) = 0 P 


It will now be proved that D® = §. By the definition of the 
class D;, there exist two and only two redundant pairs (k,m) and 
(p,q), and they have the properties: 

(kym) - (4,7) =1 

(p,q) - (4,7) =J. 
These pairs may define in total cither three or four members of the 
3-chain (three members when m = p, but no fewer for if k = p and 
m = q then (k,m) - (7,7) = (4,7), which is contrary to the definition 
of D;). Suppose m = p, then either i = y. = 7 or i= y; = 7, which 
is impossible for i #> i by assumption. Thus m # p. With four 
members there are two possibilities for a redundant 3-chain: either 
1 = ye, Ys =] OL 1 = ¥3, Yo — 7. The former is impossible by the 
previous argument; thus the only 3-chains of the class D, are of the 
form 

05925 ¥a59 =t5954,9; 

that is, 

=} if ¢<=>3 


d;;® ; 
‘  =0 otherwise. 


Therefore, by the definition of S, we have D® =S. 
If the matrices of redundancies up to and including R‘* are 
known, then we can find A™ by A™ = XR“™)X. 
Proof: By the definition of the class A, , a redundant n-chain of this 
class has the form 
: (a) (b) (c) rae 
= Vis Vay = SRE ee I a IS 
wherea+b+e+5=n,k<m, and x= yn. 
(n-2) 
It follows from the definition that p = y. ==> y, = q is a redundant 
n—2 chain, and each such distinct n—2 chain determines no more than 
one distinct redundant n-chain from i to 7. Thus the number of re- 
dundant n-chains of type A, from i to 7 is the sum over all combina- 
tions p = y. and q = v, for the number of redundant n—2 chains from 
p to q, that is, 
aijy™ — > > LigGee'* MX; 
(p)+(qQ) <= 


or 
A™M=XR”™X. 
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If the matrix [¢;;“] is defined as 
[ei;™] = X hiv X -} D™ 
then the relations 


Am + Bm + Dm = ROY 
AM +¢CM + D™ = XRX®» 
E™ = [64; (ij — ij) ] 


follow through an enumeration of cases and by using similar patterns 
of proof to that just given. 


These various relations permit the specific conclusions: 


R® = [6;;25;] = 77) 
R® = XR® + ROX + EO—S 


and the general result 


R™ = XRD) + ROUX — XR X 
+EM— pm, 


This latter expression is not useful in its present form because D™ 
has not been expressed in terms of the matrices of redundancies up 
to and including R’. This problem of the determination of the 
matrix of redundant n-chains is left as an unsolved problem of both 
theoretical and practical interest. 


5.10. Uniqueness: 


In certain applications it is desirable to know whether a power 
of a matrix uniquely determines the matrix. This is not true in gen- 
eral, for Sylvester’s theorem gives a multiplicity of n roots of a 
matrix. The matrices being considered are rather specialized, how- 
ever, and it is possible that some degree of uniqueness may exist. 


The following two theorems indicate certain sufficient conditions 
for uniqueness. Since these theorems do not utilize completely the 
special characteristics of the matrices in this study, it is probable 
that more appropriate theorems can be proved. 


5.11. Theorem 6: If p and q are positive integers, if two inte- 
gers a and b can be found such that ap — bq = 1, and if X is a non- 
singular matrix, then the powers X” and X¢ uniquely determine X . 
Proof: Suppose that there exist two non-singular matrices X and Y 
such that X*? = Y? and X’= Y?. Then X” = Y and X= Y". Now, 
form X*“Y = Yuy = yu! = YY, since ap — bq = 1. Similarly 
XX = X11 = X"”, But since X” = Y” it follows that XX = X™Y. 
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Since X is non-singular, |X°*| # 0, and thus there exists a unique in- 
verse of X°%, X-*%, such that X-""X"1 = 1; therefore X = Y . 


5.12. Theorem 7: If 1 is a positive odd integer and S a real 
symmetric matrix, then there is one and only one real symmetric n** 
root of S. 


Proof: 1. There is one such n‘ root. 


Since S is real and symmetric there exists a real orthogonal 
matrix P such that PSP = D (P’ is the transpose of P) is diagonal 
with real entries d;; which are the characteristic roots of S .* Assume 
P is so chosen that di, < dex < ---< dnm. Let B be the diagonal ma- 
trix of the real n‘* roots of the elements of D, i.e., bj; = real (di), 


so 
Br=D. (1) 


Define R = PBP’. Then R° = S, for 
R*= (PBP’)*= PB" = PDP=S. 


Since B is real and diagonal and P is real and orthogonal, R is real 
and symmetric. 


2. There is only one such n” root. 


Suppose there exists a real symmetric matrix R, not equal to R 
such that R,” = S. Then there exists an orthogonal matrix Q such 
that Q’R,Q = T is diagonal in the characteristic roots of R, , and or- 
dered as before. Consider the n power of T: 

T"= (QR,Q)*=QFR,"Q = U'SQ 
= Q'PDP'Q = (P’Q)'D(P’Q) 
7T™*=U'DU, (2) 
where U is the orthogonal matrix P’Q. Since U’ = U-, T" and D are 
similar, and hence have the same characteristic roots.+ Because they 


are diagonal in the characteristic roots, ordered in the same way, 
they are equal: 


po ™. (3) 
Substituting (3) in (2) 
D=U'DU 
or 
UD= DU. 


*MacDuffee, C. C. Vectors and matrices. Ithaca, N. Y.: The Mathematical 
Association of America, 1948, pp. 166-170. 
;Ibid., p. 1138. 
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By definition of matrix multiplication this means 


ps Uj jd jx a >. dj Ujx. 


Since D is diagonal, this reduces to 
Uj, = di iMix 
or 
wiz (di —Ai;) =0. (4) 
Since the d,, are real and v is odd, equation (4) implies 
Wik [ (Axx) /" — (dii)”"] =0 


where the (d,.)?/" are real. Thus by the definition of B, 


UB = BU 
or 
B=U'BU. (5) 
By (1) and (3) 
T= D=B", 


but by construction T and B are both real diagonal matrices and n 
is odd, so this implies 


r= B. 
This substituted in (5) gives 


T = U'BU=Q'PBP'Q 
or 


OTO = PRP’. 
But QTQ’' = R, and PBP’' = R by definition; therefore 
R.=F. 
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A NOTE ON THE ESTIMATION OF TEST RELIABILITY 
BY THE KUDER-RICHARDSON FORMULA (20)* 


LEDYARD R TUCKER 
EDUCATIONAL TESTING SERVICE 


The Kuder-Rickardson formula (20) is rewritten to be identical 
with the simplest formula, (21), except for the addition of a term 
involving the standard deviation, o,, of the item p’s. If 9, can be 


estimated, a rapid and superior estimate of test reliability is pos- 
sible in contrast to the simpler formula (21) used when the num- 
ber of items and mean and standard deviation of test scores are 
known. 


Quick estimates of test reliability are frequently desired and the 
Kuder-Richardson formula (21) is often used. This formula, how- 
ever, sometimes seriously underestimates the reliability of a test. The 
Kuder-Richardson formula (20) yields a much better estimate, but 
requires results from an item analysis of the test. Formula (20) 
here is rewritten to involve only the standard deviation, «,, of item 
p’s in addition to the number of items and the mean and standard 
deviation of test scores. 

Following the Kuder-Richardson notation, formulas (20) and 


(21) are: 
n of —2n pq 
come ot (MEY, a 
n—] or~ 


n npg 
re =( _"_) (2 : ‘) . (21) 
n—I1 or 


where 7;; is the test reliability; o,° is the variance of scores on the 
test; x is the number of items; p; is the proportion of candidates giv- 
ing the correct answer to item i; q; is (1—p:); p is the mean p;; q, 
(1—p), is the mean q;; and pq is the mean piq;. Equation (22) gives 
the relation of » to the mean score, M;, on the test: 

_M;: 


p- ae (22) 


*Kuder, G. F. and Richardson, M. W. The theory of the estimation of test 
reliability. Psychometrika. 1987, 2, 151-160. 


117 











118 PSYCHOMETRIKA 


(Equation (22) is the last one given by Kuder and Richardson. The 
formulas in this note will be numbered from (23) on for simplicity.) 
Formula (21) is derived from formula (20) by assuming that all 
items are of equal difficulty. Our approach is to rewrite formula (20) 
so that it involves n, o¢, M;, and o,. Other more realistic assump- 
tions of o, may then be made. 

A formula for a,’ is: 


of =—> 7p? —f. (23) 
N ix 
But: 
es 1 n 
pq=—D Pid, 
N j=1 
(24) 


7: n 
=—>p: (1-—p), 


i=1 


| 


=p——Spe. (25) 
nN ia 


Substituting equation (23) in equation (25): 


pq =p — p? — o;? 
= p(1—p) — o,’ (26) 
=pq—o,;? . 


Substituting equation (26) in formula (20): 


n o:? —Np q+ No,’ 
r= (— )(“ee), (27) 
n—1 \ ot” 


Or, using equation (22): 


 M;? 
n ot" == M, + + No,” 
ty=| — os > 2 (28) 
a3 or 


Equation (27) is quite similar to formula (21), since only one term, 
No,?, is added in the numerator. 

It is immediately apparent that formula (21) holds whenever o, 
is zero. However, better estimates of o, can be made if item diffi- 
culties are known. Ideally, o, is to be computed from an item analy- 
sis of the test in its final form, but estimates can be made from analy- 
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ses of the items in experimental forms of the test or in other tests. 
It might even be possible to guess a practicable value of s, from edi- 
torial judgment of the test items. The results for formula (21) are 
seldom more than 10 per cent less than those for formula (20). Con- 
sequently, the term involving o, contributes less than 10 per cent to 
the estimate of the test reliability. Thus even an error of 20 per cent 
in o,” will result in only a 2 per cent error in the estimated test re- 


liability. 
Results for two tests are summarized in the following table. 
Reliability 
Odd-Even 

Type of Test n M, a, o, Formula 21 Formula 27 (Corrected) 
Verbal ............ 140 68.5 21.5 .052 .93 .95 .95 
Mathematical 

Aptitude ... 55 19.5 4.9 .057 81 86 .87 


In these cases n was found by a count of the number of items, M; 
and o; were computed from a frequency distribution!of total scores, 
and o, was obtained from an item analysis of the test in its final form. 
Two useful types of limits for o,? are that: a) for a rectilinear dis- 
tribution of item p’s from .00 to 1.00, «, equals .083, and b) for a nor- 
mal distribution with .00 at —3oe, and 1.00 at +3e,, o,? equals. 028. 
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APPLICATION OF THE CONCEPT OF SIMPLE 
STRUCTURE TO ALEXANDER’S DATA* 


MARIANO YELA 
THE UNIVERSITY OF CHICAGO 


A battery of 20 tests originally analyzed by Alexander (1) was 
reworked according to the principle of simple structure. His results 
were sustained in general. Both analyses yielded five factors in the 
first-order domain. Of these, three factors in the re-analysis (v, X 
and F’) have almost exactly the same loadings as the corresponding 
factors in the original work, and were interpreted in the same way. 
The loading pattern of a fourth factor, Z , left uninterpreted in the 
original study, happened to be more clear in the re-analysis, and an 
interpretation was attempted. It appears to be a factor of perceptual 
synthesis, and seems to play an important role in intellectual process- 
es. A fifth factor, not present in Alexander’s results, appeared in the 
new analysis: the reasoning factor, involved in inductive and deduc- 
tive thinking. All four cognitive factors are related to a general fac- 
tor that can be thought of as representing abstraction and eduction 
of relations and correlates, these processes being, therefore, the es- 
sential feature underlying intellectual behavior, at least in that sec- 
tor surveyed by the tests of the present battery. 


Among the papers dealing with the existence of a general factor, 
Alexander’s work (1) is one of the most interesting and carefully 
done. It is, for instance, the first attempt to reconcile Thurstone’s 
methods with Spearman’s theory. 

The clarity and thoroughness of his testing and exposition, to- 
gether with the importance of his results, on the one hand, and the 
fact that we believe better techniques of rotation have been made 
available since his work, on the other, moved us to try a new fac- 
torial study of his data. 


The Work of Alexander 
The paper of Alexander presents the analysis of four batteries 
of verbal, performance, mechanical, and perceptual tests, given to 
four different experimental populations. An adequate discussion of 
the testing procedure is given in his paper, so that we do not need to 
be concerned here with this topic. 


*The analysis of the data was done in the Psychometric Laboratory of The 
University of Chicago, under a fellowship from The University of Madrid (Junta 
de Relaciones Culturales). The author wishes to express his gratitude to Pro- 
fessor L. L. Thurstone, whose advice as a scientist and kindness as a friend, 
have been the principal stimulus for this work. 
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The four correlation matrices were factored by the centroid 
method. Subsequent orthogonal rotations led Alexander to final struc- 
tures which he interpreted, arriving at five independent factors: a 
general intellective factor g , a verbal factor v, a performance factor 
F, a character factor X , which he named “persistence” or “will to 
succeed,” and a factor Z the nature of which did not seem to be 
clearly indicated in the study. 

Of the four groups studied in the original work we will be con- 
cerned here only with Group 3.7 

The battery given to this group consisted of twenty tests, as fol- 
lows: 


Achievement in Shop Work (Engineering, etc.) 
Achievement in Mechanical Drawing 
Achievement in Mathematics 

Achievement in Science 

Achievement in English 

Verbal Test 1 (Terman Group, 1, 2, 3, 4) 
Verbal Test 2 (Terman Group, 6, 7, 8, 9) 
Verbal Test 3 (Otis Self-Administering Test) 
Verbal Test 4 (Otis Group, 1, 2, 3, 4, 7, 8) 

10 Number-Verbal Test 1 (Terman Group, 5 and 10) 
11 Number-Verbal Test 2 (Otis Group, 5 and 6) 
12 Cox Test E3 

13 Cox Test C 

14 Cox Test D 

15 The Passalong Test 

16 Kohs’ Block Design Test 

17 The Cube Construction Test 

18 Spearman’s Test 1 (Form Series Test) 

19 Spearman’s Test 2 (Dot Pattern Test) 

20 Spearman’s Test 3 (Form Analogies Test) 


CANA hrWwWhN eH 


Discussion of these tests and references to the literature can be 
found in the original paper. 

Alexander’s method led him to the rotated orthogonal matrix 
given in Table 2. The factors represented in this table were defined 


by Alexander as follows: 


+Groups 1, 2, and 4 were analyzed by Alexander into three factors: g, 1, 
and F. We factored and rotated to simple structure these correlation matrices 
and found three factors in each case: v, F, and a factor which could be called 
induction or reasoning. These results are not reported in full because they are 
in agreement with the results obtained in the analysis of Group 3, which will be 
fully reported, and because, having only three primary factors, we cannot have 
any assurance in the study of the second-order domain. 
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A general factor g. As this factor was common to all 
tests in the battery and as its corresponding axis was lo- 
cated through one of Spearman’s tests of g, it was iden- 
tified with the general factor of Spearman, and was de- 
fined as a general intellective factor. 


A verbal factor v. This factor was loaded in all verbal 
tests and absent from all others. Factor v was defined as 
that factor independent of g which is found in verbal tests 
of intelligence. 


A performance factor F. This factor was most conspicu- 
ous in the Cube Construction Test, with a loading of .70, 
and had high saturations on Cox’s tests and lower satura- 
tions on Block Design, Passalong, and Shop Work. It was 
defined as that factor independent of g which is found in 
performance tests of intelligence. 


A temperament or character factor X, present in all 
school achievement tests and absent from all others. It 
was defined as the factor present in school achievement 
and independent of g and v , and was named “persistence” 
or “will to succeed.” 


A factor Z, uninterpreted but proposed as related to 
school achievement. 


Discussion of Alexander’s Method 

From the viewpoint of multiple-factor analysis as it is generally 
understood today in America, one of the important merits of Alex- 
ander’s work is the fact that he realized, as early as 1935, the impor- 
tance of rotation in the factorial procedure. 

As a result of factoring the correlation matrix one has an or- 
thogonal reference frame which is arbitrary in the sense that the lo- 
cation of the frame in the test space depends on the mathematical 
method of factoring adopted. Alexander states that this statistical 
frame has to be rotated to a psychological frame before the interpre- 
tation can be undertaken. By this he means that the axes should be 
placed in the location of the greatest psychological significance. And, 
he adds, a vital point is that the axes should be orthogonal or uncor- 
related, because we are looking for independent factors. How do we 
know, however, that a certain frame has the greatest psychological 
significance? 

The criterion adopted by Alexander is to pass the axis through 
a chosen vector. Thus he needs, of course, another criterion for the 
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selection of the test to be used as pivot. He goes then to previous 
findings in factorial studies. He chose one of the tests offered by 
Spearman and Stephenson as a good measure of g, and passed the 
first axis through it. Another test already identified as a good meas- 
ure of v was chosen as the pivot for the verbal factor, and the sub- 
sequent axes were drawn through the clusters of residuals left after 
the effect of the previously identified factors had been ruled out of 
the tests. 

Now, we cannot avoid the suspicion that if we use our previous 
hypothesis or the result of previous experiments as a criterion to lo- 
cate the frame, we run the risk of forcing the results to be in ac- 
cordance with our expectations. If the results turn out to be psycho- 
logically meaningful, this can be due to the correctness of our hy- 
pothesis or to the fact that they agree with what we previously 
thought of as a meaningful psychological theory because we have used 
this psychological theory as a directing criterion in our work. 

What we need in factorial analysis is a criterion independent of 
our particular theory and the findings of previous factorial studies. 
We do not need to be concerned beforehand with the problem of which 
tests would give more meaning to the frame. We need first of all a 
criterion as independent as possible of any specific theory and as de- 
pendent as possible on the properties of the test configuration itself. 
The ideal would be to discover the structure that is demanded by the 
configuration, instead of imposing an assumed meaningful frame 
upon the configuration. 

There is an infinite number of structures corresponding to an 
infinite number of reference frames. Mathematically any frame and 
consequently any structure is as good as any other. But perhans there 
is a particular reference frame which is especially connected with 
the configuration, such as to be strongly demanded by the configura- 
tion itself. This is the assumption underlying the concept of simple 
structure (Thurstone, 2, 319-346). This assumption only goes as far 
as to gamble on the existence of planes well defined and overdeter- 
mined in the test configuration and by the test configuration itself. 
In Thurstone’s own words: “If an analysis is to be made by the prin- 
ciples of simple structure, then the investigator gambles that the com- 
plexity of each test or measurement is less than the complexity 7 of 
the battery as a whole.” (2, 320). Furthermore, it is believed that 
the appearance of simple structure is not a matter of chance (2, 328). 
Both statements spring from the basic assumption: “In the interpre- 
tation of mind we assume that mental phenomena can be identified in 
terms of distinguishable functions, which do not all participate equal- 
ly in everything that mind does.” (2, 57). 
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In any particular investigation it is, then, a matter of fact wheth- 
er the simple structure is present or not, and whether the discovered 
simple structure can or cannot be meaningfully interpreted, or, if in- 
terpreted, whether it agrees or not with our previous hypothesis or 
with the findings of previous studies. In this light we do not need to 
impose the condition that the axes should be orthogonal. Whether 
they are orthogonal or oblique depends on the configuration. With 
this we do not lose the independence of our factors. It is necessary 
to distinguish between linear and statistical independence. We want 
to know the dimensionality of the psychological structure. Whether 
the parameters which represent these dimensions are statistically cor- 
related or not is a fact to be found out rather than to be postulated. 

All this does not mean that any other method of rotation is nec- 
essarily wrong. The literature shows how different methods have ar- 
rived at comparable results. Alexander’s assumptions seemed to us 
quite reasonable, so that we believed from the beginning that our re- 
sult would not differ greatly from his, except perhaps in the general 
factor. What we mean is that, in our opinion, the method of simple 
structure is, among those now available, the most rigorous and ob- 
jective to check a previous hypothesis and the most flexible to explore 
a new domain. 


Results of This Study 


The centroid matrix F,, given in Alexander’s paper (1, 109; our 
Table 1), was rotated to simple structure. Table 3 shows the obtained 
oblique factor matrix V. To facilitate comparisons we have placed 
Alexander’s final matrix and ours together (Tables 2 and 3), as well 
as the corresponding factor patterns (Tables 5 and 6). Table 4 gives 
the transformation matrix A. 

It can be seen that our loadings are almost exactly the same as 
those found by Alexander for the four factors present in both ma- 
trices. To make the comparison easier we have reversed axis Z in 
Alexander’s structure. 

Within the limits imposed by the high complexity of some cf the 
tests, the structure here presented satisfies the standard requirements 
for simple structure (2, 335 ff.). 

Factors v , F, and X have the same pattern and lead to the same 
interpretation in both studies. Factor Z shows a more clear pattern 
in our study as a result of the application of the concept of simple 
structure. 

The interpretation of the factors is as follows: 
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FACTOR Z 
Principal saturations on factor Z 
Code number Name Factor loading 
16 Kohs’ Block Design AT 
19 Dot Pattern Test AT 
18 Form Series Test 39 
15 Passalong Test 27 


All these tests require the subject to form or complete a configu- 
ration. This factor seems to represent speed of closure of a pattern 
to be formed following some formal instructions and against conflict- 
ing or irrelevant elements. 

The subject will excel in this task if he can hold the given struc- 
ture as a group of elements organized into a pattern and at the same 
time can reproduce it quickly (Tests 16, 19, and 15), or is able to per- 
ceive the figure that completes the unfinished configuration (Test 18). 
At the beginning of the task the elements integrate themselves into 
changing configurations that interfere with the completion of the 
correct pattern. In all cases the final structure is arrived at by quick- 
ly rejecting the patterns that do not lead to the correct configuration 
and by the ability to synthesize the units given into a meaningful 
whole. 

This factor is similar to some others revealed in the recent lit- 
erature. Thurstone (3) found a factor of perceptual closure (factor 
A), and another of flexibility in manipulating several irrelevant or 
conflicting “Gestalts” (factor E). In both factors the Block Design 
Test had significant saturations. In a battery of perceptual tests Bech- 
toldt (4) found a factor G , speed of closure of 2 visual configuration 
similar to our synthetic factor; he also found another factor Y similar 
to Thurstone’s E , which he defines as a facility in organizing several 
simultaneous or successive configurations into a larger pattern under 
the distraction of further activity. Other studies such as those of 
Meili (5) and Rimoldi (6) present similar factors. Rimoldi reports 
a factor B as involved in the perception of relations in space neces- 
sary for the construction of a whole, and a factor C as a facility in 
solving the conflict between two or more configurations. These factors 
are related to those called “globalization” and “plasticité” by Meili. 
Factor Z is probably a composite of these two types of synthetic fac- 
tors, which are not necessarily opposite to each other, as Meili pointed 
out (5, 43). Furthermore, in these studies there were found some 
interesting relations between this synthetic factor or factors and rea- 
soning. Thurstone (3) reports a correlation of .39 between a com- 
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posite test of factor A and another of factor R (reasoning). Also he 
found that the reasoning tests had a saturation of .42 on factor EF’. 

Bechtoldt reports a closure factor of second order, tentatively 
interpreted as a facility in forming conceptual structures, in which 
two primaries had loadings: the perceptual closure factor G and a 
factor of speed of ideational closure with verbal materials V. In our 
study the primary Z shows a correlation of .59 with R (reasoning), 
as shown in the correlation matrix R (Table 7). Both primaries Z 
and F have the highest saturations in the second-order general factor 
(Table 10). These results present some additional evidence to the 
findings of recent research in pointing out the importance of a closure 
or synthetic perceptual factor in the performance of tests of intelli- 
gence. 

Several theories and clinical observations have in the past sug- 
gested the existence of such a trait. We should remember in this con- 
nection the theory of the synthetic sense of Aristotle and the scholas- 
tic psychologists (7), and the contention of the Gestalt theory that 
the ability to form “Gestalts” and the freedom from “Gestaltbindung” 
are fundamental features in productive thinking. 

Examination of the other tests in the battery shows that this per- 
ceptual closure is not specially required in their solution. A percep- 
tual synthesis of the material may be necessary in all thinking, but 
only in those tests demanding special synthetic ability would it be 
responsible for individual differences. There are, however, two tests 
that at first sight seem to depend directly on this ability. These are 
the Cube Construction and the Form Analogies Tests. 

In the Cube Construction Test (cf. 1, 42 and 154 f.) the subject is 
required to reproduce a model by manipulating items in a way simi- 
lar to that demanded by Tests 15 (Passalong) and 16 (Kohs’ Blocks). 
A closer study of the tests and the subjects’ performances would 
show, however, that in Tests 15 and 16 the subject works with the 
elements as pieces related to one another forming a whole. He is 
looking for the best way to arrive at a configuration. He perceives 
not independent elements, but partial configurations, and these he 
perceives related to the remaining items until a total closure is at- 
tained. This is clearly so in Kohs’ Blocks and to a lesser extent in the 
Passalong Test. Accordingly, Test 16 has a loading of .47 and Test 
15 a loading of .26 in the factor. 

In Test 17 (Cube Construction), on the contrary, the items of 
the model are partially hidden. To see the model as a configuration 
and to fit the items together, the subject has to visualize in his mind 
the different possible positions of the blocks. He is likely to work 
with each item separately, especially in the difficult models, and place 
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it without a clear idea of the total structure. If this description is 
right we would expect this test to have a high saturation in some 
spatial factor. Actually we have a space factor in the analysis, and 
this test has a loading of .70 on this factor. Tests 15 and 16 require 
evidently this sort of visualizing of the figures as they change in 
space, and accordingly they also have some loading in this factor 
(.25 for Test 15, and .29 for Test 16). 

The scoring of these tests supports this interpretation. Tests 15 
and 16 are scored as a whole. The task is right or wrong and the 
subject gets a score for the time consumed in the successful perform- 
ances. In Test 17 the score is a function of time and success per unit; 
i.e, the task is broken up in little tasks and each block correctly 
placed is considered as a success regardless of the position of the 


others. 

The Form Analogies Test (number 20) is the only one among 
the remaining tests in which some element has to be picked up out 
of some irrelevant ones to closure an incomplete perceptual configu- 
ration. This test has a loading of .19 on the synthetic factor. This 
loading is probably insignificant, so that our results do not warrant 
any further discussion of it. 

One question regarding this factor remains. Test 1 (Shop Work 
Achievement) has a negative loading on this factor (—.38). It is 
not clear how the facility to integrate diverse elements into a con- 
figuration would harm shop work performance. This test also has a 
high negative saturation in the corresponding factor of Alexander. 
All other tests in our configuration are included within reasonable 
limits in a positive region. And then, Test 1 stands out alone. This 
may be due to some computational error. Or maybe it is due to some 
actual characteristic of the variable. That we cannot decide from the 
information given in the original paper. 


FACTOR R 
Principal saturations on factor R 
Code number Name Factor loading 
10 Terman Group Test (Subtests 5 and 10) 46 
11 Otis Group Test (Subtests 5 and 6) 52 
20 Form Analogies .o7 


Test 10 is composed of arithmetic problems and geometric figures 
(a circle, a triangle, and a rectangle with numbers and directions). 
Test 11 consists of arithmetic problems and number series. Test 20 
requires the subject to choose among several forms the one related to 
a second as a third is to a fourth. The common feature involved in 
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the performance of these tests is the ability to find out a general prin- 
ciple by analysis of the given elements or the application of a rule to 
solve a problem or to identify an element. Some tests, as the mathe- 
matical problems, involve generally both processes, which can be 
called inductive and deductive thinking. Accordingly we may call this 
a reasoning factor. The complexity of Tests 10 and 11 may be respon- 
sible for the fact that we find a single factor for inductive and deduc- 
tive reasoning, since some indications of two separate factors have 
been reported in the literature. Or it may be that both processes can 
be a function of a single factor. 

Other tests in the battery have lower loadings in this factor. Test 
8 is the only verbal test with an appreciable loading (.26) in this fac- 
tor. Inspection of the test shows that it is the only verbal test having 
among its items a number of arithmetic problems and the same sub- 
test of geometric figures as Test 10. 

Form Series has a loading of .21 on the reasoning factor. This 
test had also, it will be remembered, a loading of .39 on the synthetic 
factor. Examination of the test shows that it can be solved in two 
ways or that two factors are required in its performance. The fourth 
form which continues the series initiated by the first three forms may 
be found out by closure of the unfinished configuration, perceiving the 
lacking element as connected with the others, or by analytically rea- 
soning out the principle connecting the items. These processes have 
been recognized by Spearman to be present in the solution of the 
Raven matrices (8), and have been called by him the synthetic and 
the analytic approach respectively. Spearman states that the ana- 
lytic procedure, not the synthetic, tends to load noegenetic processes 
with g. The analytic procedure would tend to load the performance 
with RF, in our factor pattern. The synthetic procedure would tend 
to load the performance with Z. It is likely that both processes are 
used by most subjects in the task. 

Tests 12 and 13 have saturations of .24 and .21 in this factor. 
They are the Cox Tests E3 and C. They are proposed as measures of 
mechanical ability. They require the subject to find out solutions to 
mechanical problems presented by drawings. They require the sub- 
ject to put into practice the mechanical principle involved in each 
problem together with the ability to visualize the position of the dif- 
ferent pieces as they move in space. This is consistent with the fact 
that both tests have also a loading of .40 on the space facter. 

Test 1 has a saturation of .35 on this factor. It has also a load- 
ing of .36 on the space factor. If we assume that Shop Work Achieve- 
ment, together with the Cox Test, is a measure of mechanical abil- 
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ity, this ability would be a function of the space and reasoning fac- 
tors, since these tests have saturations in both. 


FACTOR F 
Principal saturations on factor F 
Code number Name Factor loading 
17 Cube Construction -70 
12 Cox Test E3 40 
13 Cox Test C 40 
14 Cox Test D zt 
1 Shop Work Achievement 36 
16 Block Design 29 
15 Passalong 25 
2 Mechanical Drawing Achievement .20 


This factor has the same loadings as factor F of Alexander. He 
interpreted it as a performance factor. Considering more recent fac- 
torial literature, we can conclude that this factor is the space factor 
S , as indeed has been already suggested (cf. Carroll, 9, 308). 


FACTOR V 
Principal saturations on factor V 
Code number Name Factor loading 
6 Verbal Test 1 -70 
5 English Achievement -70 
7 Verbal Test 2 .66 
9 Verbal Test 4 59 
4 Achievement in Science 59 
8 Verbal Test 3 44 
3 Achievement in Mathematics 45 


This factor is readily identified with the verbal comprehension 
factor, and Alexander’s interpretation is sustained. Perhaps it should 
be pointed out that since Alexander’s work a verbal fluency factor W 
(Thurstone, 10) and an ideational fluency factor F (Taylor, 11) have 
been discriminated as different from the verbal comprehension fac- 
tor V. 


FACTOR X 


In this factor only the achievement tests have saturations (.76, 
.65, .55, .49, .42). Our results agree fully with those of Alexander 
so that we do not need to add anything to the discussion and inter- 
pretation of this factor presented in pp. 125 f. of his paper. 
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The Second-Order Domain 

Factorization of matrix R (Table 7), resulted in the factor ma- 
trix F, (Table 8). 

Since the reference axes m, are arbitrary and no simple struc- 
ture was over-determined by the five primaries, we rotated the axes 
so that the five primaries would be included in a positive quadrant, 
on the assumption that the second-order factors would not be nega- 
tively correlated with the first-order primaries. The oblique axes are 
shown in matrix V. (Table 10). Table 9 gives the transformation 
matrix for the second-order factors. 

A factor appears in the second-order domain common to all cog- 
nitive primaries. The loadings are as follows: reasoning .76, per- 
ceptual synthesis .71, verbal .59, space .50, character factor X .00. 
This factor seems to be independent of X , in which we see a corrobo- 
ration of Alexander’s belief that factor X does not belong to the field 
of cognition. The structure of the second-order domain is not stable 
enough to attach much confidence to these results, but the correspon- 
dence with Alexander’s findings is worth noticing. We do not believe 
that from this study nor from the total factorial research done so far 
a final answer can be presented as to the nature of the general intel- 
lective factor. If something is common to all cognitive functions, the 
perceptual synthesis, the verbal comprehension, the manipulation of 
space images, and the inductive and deductive thinking, to consider 
only the field covered by Alexander’s battery, this would be the ca- 
pacity of the subject to understand his task and his working, i.e., to 
see the meaning of the words, percepts, spatial relations, and logical 
relations, and to see the relation of these meanings to one another 
and to the solution of the problem. This general feature in all intel- 
lectual tasks has been named by Spearman “abstraction and noegene- 
sis” (8). This explanation seems logical. The validity of this expla- 
nation is difficult to ascertain from the factorial studies alone. We 
believe, however, that the factorial way of attacking the problem is 
the analysis of batteries with cognitive and non-cognitive tests, so 
constructed that a second-order domain can be expected to have a 
simple structure as overdetermined as those so far found in the first- 
order domain. If a general intellective factor is present, as a great 
deal of evidence indicates, its nature will be better understood when 
we know which factors are related to it, which are independent, and 
how much the related factors are loaded in the general factor. By 
the application of the rotational principle of simple structure we, 
then, were able to get the same information as Alexander, plus some 
interesting new items, as reported above. This we take as supporting 
our belief that Thurstone’s principle of rotation affords a more fruit- 
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ful approach to the problem and, contrary to some opinions, does not 
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preclude the finding of a general factor. 














TABLE 1 

Alexander’s Centroid Matrix F* 
I IT Il IV V h2 
1 403 187 350 064 515 589 
2 302 276 575 —057 063 776 
3 660 353 244 —186 062 658 
4 620 505 291 —040 —008 726 
5 581 574 —023 —056 —083 678 
6 715 884 —422 189 —190 873 
7 700 221 —416 73 —293 828 
8 820 052 —326 012 —032 784 
9 767 184 —318 —046 —182 759 
10 785 015 —172 —273 157 745 
11 706 —098 -—148 —282 221 658 
12 651 —205 —075 194 080 515 
3 53 316 —I177 243 022 480 
14 5381 —215 054 069 001 336 
15 458 —213 067 128 —175 307 
16 540 —458 141 084 —257 594 
17 431 —206 094 565 098 565 
18 455 —387 089 —307 —098 469 
19 414 —335 177 —186 —238 406 
20 568 —282 —061 —264 059 479 
*Decimal points have been omitted from this and suhse- 


quent tables. 
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TABLE 2 TABLE 3 
Alexander’s Rotated Our Oblique Matrix V 
Orthogonal Matrix Z X V F R 


— es Ue 385 489 -027 358 349 
447. 503 O11 278 248 069 760 313 204 -022 
000 766 4173 179 358 000 546 448 009 112 
-118 563 351 -020 4652 034 650 574 070 058 
095 633 477 051 292 076 420 701 -073 -049 
-152 394 656 -082 259 040 -008 700 100 002 
065 822 122 402 095 -067 663 082 -057 
068 -124 779 128 481 022 -047 435 126 259 
-114 -055 548 128 673 107 016 587 -015 104 
000 000 646 000 585 10 012 061 267 -015 462 
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-219 098 276 -060 730 11 -022 005 115 012 6521 
ll -232 065 153 -026 759 12 023 -017 079 399 242 
12 000 -029 211 398 558 18 063 -187 -001 395 210 
18 032 -205 164 408 496 14 148 056 042 272 152 
14 063 060 091 277 493 15 269 045 104 250 -034 
15 235 0380 185 281 393 16 473 -004 -014 2938 -023 
16 401 -004 -003 339 564 17 -071 061 -041 696 026 
17 000 «=©000)=— 141 70400 221 18 393 018 -072 -040 211 
18 219 043 -1388 -027 632 19 474 066 009 015 006 
19 354 098 -076 050 512 20 186 -057 -006 010 370 
20 000 «6000 )«=6—000 Ss 000—s«G9!. ial leah ecainamatie | ae peices 

TABLE 4 





I 122 287 396 270 = 251 
IIT -3893 525 7389 -265 -265 
III 247 817 -167 248 -296 
IV -265 -020 007 37 -328 
V -8386 020 -518 308 820 
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TABLE 5 TABLE 6 
Alexander’s Factor Pattern Our Factor Pattern 
Z X V F g Z xX V F R 
16 40 34 56 16 47 29 
19 35 51 19 47 
18 22 63 18 39 21 
15 24 28 39 15 27 25 
1 —45 50 28 24 1 -88 49 36 35 
2 a7 36 2 76 31 20 
3 56 35 45 3 55 45 
4 63 48 29 4 65 57 
5 39 66 26 5 42 70 
6 82 40 6 70 
7 78 43 cf 66 
8 55 67 8 44 26 
9 65 58 9 59 
17 70 22 17 70 
12 21 40 56 12 40 24 
13 —20 Al 50 3 40 21 
14 28 49 14 af 
1 —22 28 73 10 A f 46 
11 —23 76 11 52 
20 69 20 37 
TABLE 7 
Correlations between 
Primaries, R 
— a x V =. 
=<  _ tee eee 
V .228 -—.258 1.000 
X .083 1.000 
F 346 —.178 392 §=1.000 
R .168 461 .258 1.000 
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TABLE 8 TABLE 9 
Orthogonal Transfor- 
Factor mation 
Matrix F, Matrix A, 
I, a, G WwW 
Z 608 488 i 953 216 
xX -190 558 II, 805 977 
V 687 —-200 
F 563 -120 
R 651 440 
Table 10 
Factor 
Matrix V, 
G Ww 
Z 718 559 
xX = -0O11 504 
V 594 -047 
F 500 004 
R 755 570 
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DEVELOPMENT OF A METHOD FOR INCREASING THE 
UTILITY OF MULTIPLE CORRELATIONS BY 
CONSIDERING BOTH TESTING TIME 
AND TEST VALIDITY* 


W. F. LONG AND IRVING W. BURR 
PURDUE UNIVERSITY 


A modification of the Wherry-Doolittle test selection method is 
presented by which tests are included in a multiple correlation (ob- 
tained for a given battery of tests) in the sequence in which the rate 
of return in validity per unit of testing time is greatest, rather than 
in the order of the size of their contribution to the multiple corre- 
lation. It is proposed that the modified method can be utilized prof- 
itably when there are economic or practical limits on the time avail- 
able for test administration. 


Introduction 

In situations where selection of personnel is based, at least in 
part, on test results, multiple correlation coefficients are being used 
more and more extensively. Such a multiple correlation is a meas- 
ure of the relationship between a criterion and a battery of tests. 
After a multiple correlation is obtained, a multiple regression equa- 
tion is calculated from which the criterion can be predicted with the 
highest precision of which the given battery of tests is capable when 
using a first-degree equation. 

Possibly the best available method for general use for obtaining 
such multiple correlations is the Wherry-Doolittle test selection meth- 
od.+ It makes possible the selection from a number of tests of the 
battery or team of tests which gives the maximum possible multiple 
correlation with a minimum number of tests, and at the same time 
saves an appreciable amount of statistical work in comparison with 
other methods. The obtained multiple correlation is the best estimate 
of the R for the population from which the sample was drawn, or in 
other words, it is corrected for the tendency of correlations obtained 

*The major portion of this article is based upon a thesis by W. F. Long di- 
rected by Dr. Joseph Tiffin with the counsel of Dr. Irving W. Burr. This thesis 
was submitted in partial fulfillment of the requirements for the degree of Mas- 


ter of Science in Psychology, Purdue University, June, 1947. 


+See Garrett, Henry E. Statistics in psychology and education. New York: 
Longmans, Green & Co., 1947, pp. 485-451 or Stead, W. H., Shartle, D. L. and 
associates. Occupational counseling techniques. New York: American Book Co., 


1940, Appendices 5 and 6. 
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from samples to be larger than the correlation existing in the total 
population. 

One of the convenient features of the Wherry-Doolittle test se- 
lection method is the fact that tests are selected for inclusion in the 
multiple correlation in the order of their contribution to the corre- 
lation. For example, after the first test is selected, the test with the 
highest residual validity is selected as the second test to be added 
to the battery. Next, the test with the then highest residual validity 
is selected as the third test, and so on, until the multiple R ceases 
to increase in size by an amount greater than the chance error intro- 
duced by the tests included. The term “residual validity” is applied 
to the remaining validity that a test has after the effect of the inter- 
correlations of that test with other tests is discounted as the battery 
is formed. 

Purpose.—In the application of test selection batteries, other fac- 
tors besides validity are of considerable importance, such as ease of 
administration, cost of tests, and testing time required. Of these fac- 
tors, testing time required is especially important from an economic 
point of view. 

Accordingly, it was considered desirable to develop a procedure 
in which testing time as well as validity of individual tests is con- 
sidered when tests are evaluated for inclusion in a multiple correla- 
tion, and thus for eventual use in a selection test battery. 

The procedure developed is a simple modification of the Wherry- 
Doolittle test selection method designed so that tests are included in 
the multiple correlation in the sequence in which the rate of return 
in validity per unit of testing time is largest, rather than in the or- 
der of their contribution to the multiple. 

Both the Wherry-Doolittle method and the modified method have 
been applied to the same battery of fifteen tests in order to permit 
comparison of the two methods and to furnish an application of the 
latter method. A presentation of the development of the modified 
method and a detailed outline of the procedure for its use will be fol- 
lowed by a brief description of the demonstration battery, a descrip- 
tion of the applications of the modified method, and a comparison of 


the results obtained. 


Development of Modified Method 
In order to determine which test will furnish the greatest return 
of validity per unit of testing time, the formula 


ois (1) 





Tecaz) = 





Vat (a— @)rar. 
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which is based on the Spearman-Brown phophecy formula,* is used.} 
This formula must be modified so that predicted validities for dif- 
ferent lengths of a test can be calculated when the given validity of 
a test is a residual validity and not the original validity of the test, 
as yet unaffected by its correlation with other tests. The formula 
then becomes 

Lz 


a 





(2) 





T c(az)R — 





Va+t (a?—a) Tan 
In these two formulas: 


1 c(az) = Validity of test X (correlation between test X and cri- 
terion) the length of which, in terms of time, has been 
multiplied by a factor a. It is important to note that in 
using a time value for a, it is assumed that the time for 
the test is changed in proportion to the change in test 
content, i.e., a longer test would include more items with 
the same ratio between time and number of problems. 

7g = original validity of test X . 


71; = original reliability of test X . 


Tc(az)r == Yesidual validity of test X, the length of which has 
been multiplied by a factor a. 

7 2 i 

a square of the residual validity of test X. This value is 

* what remains of 7,,” (square of the validity of test X) 

after the effect of the intercorrelations of test X with 

the other tests is discounted as the battery is formed. 


Tire — residual reliability of test X . 


The value 7::z can be estimated in terms of the original reliabil- 
ity of the test and the residual validity of the test. This can be cal- 
culated by use of the formula 








rir a —, Vex 
z 
i | Si . (3) 
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*Use of this formula assumes that the entire test is homogeneous. 


+Peters, Charles C. and Van Voorhis, Walter R. Statistical procedures and 
their mathematical bases. New York: McGraw-Hill Book Co., 1940, p. 196. 
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The derivation of this formula is not complicated: 





2 
J aE < sper oe era on original validity of test X be- 
sf cause of the intercorrelations of the test with 
tests already included in a battery. 
134;= original reliability of test X . 


71,7 = proportion of variance in a second equivalent 
form of the test (for example) explained by the 
first form. 


(1—7,,°) = (error)? or 7, + (measurement error)? = 1? 
(perfect reliability). 


If much of the correlation of test X with the criterion is ac- 
counted for by other tests already included in a battery, the reliable 
part of test X which has not yet been included may not have as large 
a reliability coefficient as the total test had. Let it be assumed that the 
equivalent of several perfectly reliable items are taken from test X , 
leaving all the factors making for error still in the test. Now it can 
be determined how much the reliability of the original test could 
have been decreased. 








Thus 
9 Ve? 
Yur a (r.. Ses -<-) 
Tsun? = V2 = (4) 
JS leas (re 3 ) aa Sasa ry 
Then 
wok: “aes 
net — — Pex? 
‘us = = R (5) 
V,? 
\ Z, 


Reference to Table 4 will show that this correction is very small 
when applied for the demonstration battery, the largest being .0025. 
Since the corrected reliability values obtained by using this formula 
are so little different from the original values and do not enter into 
the actual calculation of the multiple correlation, it may well be that 
this step may prove to be unnecessary. 
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Modified Wherry-Doolittle Test Selection Method 
A detailed procedure for use of the modified Wherry-Doolittle 
test selection method is outlined here in a general form so that it can 
be readily applied for the calculation of a multiple correlation for 
any specific battery of tests. It will be noted by those familiar with 
the Wherry-Doolittle procedure that the modified method differs from 
the original only in those steps involved in the determination of which 
test should be first included in the battery and the sequence in which 
the remaining tests should be added. These differences first appear 
in steps 4 and 10 of the modified procedure. 
Given: a. Intercorrelations of all tests.* 
b. Reliabilities of ali tests. 
ce. Correlation of the tests with a criterion (validity). 
d. Testing time for all tests. 


These data can be handled most effectively if presented in a form 
similar to Table 1. 

1. Prepare worksheets similar to Tables 2 and 3. 

2. Enter in the V, row in Table 2 the validity coefficients of all 
tests with signs reversed. 

3. Enter in the Z, row in Table 3 the number 1 for each test. 

4. Select as the first test in the battery that test which will 
give the greatest rate of return in validity per unit of test- 
ing time. This is accomplished by completing the following 
steps: 


a. Prepare a worksheet similar to Table 5. 
2 


V. 
b. Choose the test which has the largest quotient ; ; 


1 
c. Calculate time-adjusted validity values for this test and 
all tests requiring less testing time so as to permit com- 
parison of these tests matched in regard to testing time 
with the shortest test, or at other strategic testing times. 
(Explained in detail on page 149). This is accomplished 
by application of the formula 





eg 
a 
Z: 


Va + (@?—4) Tim 





(2) 





Te(ar)R — 


*Actually, only the intercorrelations of those tests included in the multiple 
correlation are needed. However, it is probably as economical in time and effort 
in the long run to calculate all intercorrelations at one time, unless the number 
of tests under consideration is large. 
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in which a is the new length of the test and 7.2 is the 


reliability of the test. The capital letter, R, generalizes 
7 2 


Vz 
the formula to make it applicable when —— and fiz are 


residual values after the effect of inclusion of cther tests 
in the battery is discounted. 


In the worksheet just prepared: 


(1) Enter in the series column, the series number. 

(2) Enter in the test number column the test numbers 
of all tests necessary to be considered. 

(3) Enter in the testing time column the testing times of 
all tests to be considered. 


7 2 
1 


of each test to 





(4) Enter in Column A the quotient 


be considered. 

(5) Enter in Column C the multiplication factor neces- 
sary to make the testing time of each test equivalent 
to the time of the shortest test. 


Then for each test in turn except the shortest: 


(6) Enter in Column F the reliability of each test. 

(7) Record in Column B the square root of the Column 
A entry. 

(8) Record in Column D the product of the Column B 
and Column C entries. 

(9) Record in Column E the difference between the square 
of the Column C entry and the Column C entry. 

(10) Record in Column G the product of the Column E 
and Column F entries. 

(11) Record in Column H the sum of the Column G and 
Column C entries. 

(12) Record in Column J the square root of the Column 
H entry. 

(18) Enter in Column K the quotient of the Column D en- 
try divided by the Column J entry. This is the time- 
adjusted validity value (the predicted validity of the 
test in question for a different testing time). If the 
time-adjusted validity of the shortest test is equal to or 
greater than the time-adjusted validities of the other 
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tests considered, it should be selected for inclusion 
in the battery. If this is not true, all tests should be 
equated to make their testing times equal to the test- 
ing time of the second shortest test. If either the 
shortest or second shortest test then has the highest 
time-adjusted validity, that test should be selected 
for inclusion in the battery. If not, this process must 
be repeated in the indicated sequence until it is ap- 
parent which test will give the greatest rate of re- 
turn in validity per unit of testing time. After a 
very few such calculations have been made, it is rela- 
tively easy to anticipate which test is most likely to 
have the highest time-adjusted validity value at a 
given time, and thus selection of the tests is not so 
complicated as it may seem. (Reference to the ex- 
planation of this step on page 149 as applied to the 
demonstration battery will facilitate its ready appli- 
cation). 


5. Apply the Wherry shrinkage formula, 
N-1 ) 
N—M 





R=1 — K+( 
in which R is the shrunken multiple correlation coefficient, 
ne pe 


and M is the number of tests so far included in the battery. 


, N is the number of subjects in the sample, 





This is accomplished by completing the following steps: 


a. Prepare a worksheet similar to Table 6. 
b. Enter in the test number column the number of the se- 
lected test. 


c. Enter in Row O, Column C the number 1. 
2 





d. Enter in Row 1, Column B the quotient 


1 


e. Record in Row 1, Column C the difference between the 
Row O, Column C entry and the Row 1, Column B entry. 

N-—-1 

N—M 

g. Record in Column E the product of the Column C and 
Column D entries. 





f. Record in Column D the quotient 
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10. 
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h. Record in Column F the difference between 1 and the Col- 
umn E entry. 

i. Record in Column G the square root of Column F. This is 
the shrunken multiple correlation coefficient. Since the 


shrinkage factor, vom , is unity when M, the number 


of tests in the battery, equals 1, the first R equals the co- 
efficient of correlation between the selected test and the 
criterion. 


Next, use the Doolittle method for solving normal equations. 


This is accomplished by completing the following steps: 

a. Prepare a worksheet similar to Table 7. 

b. Leave the a, Row blank. 

c. Enter in the b, Row the correlation coefficient of the first 
selected test with every other test, as well as with the cri- 
terion. The sign of the latter correlation coefficient must 
always be reversed in this table. 


.d. Record in the check sum column the algebraic sum of the 


entries. 
e. Record in the c, Row the product of each b, entry and 
the negative reciprocal of the b, entry for the selected 


--] 
test. Formula: c, = b, (each test) X _ (selected test) . 
Draw a vertical line under the first selected test in Tables 
2 and 3. 


To each V, entry for the tests in Table 2, add algebraically 
the product of the b, entry in the criterion column and the 
c, entry for each of the other tests (from Table 7) to obtain 
the V, entries. Formula: V, = V, + [b, (criterion) X ¢, 
(each test) ]. 


To each Z, entry for the tests in Table 2 add algebraically 
the product of the b, and c, entries of the corresponding tests 
(from Table 7) to obtain the Z, entries. 

Formulas: Z, = Z, + [b, (each test) X c, (same test) ] 


Select as the second test in the battery the test which will 
give the greatest rate of return in validity per unit of 
testing time. The selection of the second test is accomplished 
exactly as in Step 4 except that the residual reliability of 











those tests entered in Table 5 must be determined. The 
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2 
2 





2 


value of the test so selected is a measure of the amount 
which the second test contributes to the squared multiple 


correlation coefficient, R?. 


The calculation of the residual reliabilities is accomplished 
by completing the following steps: 


a. 
b. 
e. 


. Enter in Column C the 


Prepare a worksheet similar to Table 4. 

Enter in the series column the series number. 

Enter in the test number column the test numbers of all 
tests to be considered. 


. Enter in Column A the original reliability of each test. 


2 


values of each test. (The 





subscript always is the same as the series number). 


. Enter in Column E the original validity for each test. 
. Record in Column B the square of the Column A entry. 
. Record in Column D the sum of the Column B and Col- 


umn C entries. 


i. Record in Column F the square of the Column E entry. 


Record in Column G the difference between the Column D 
and Column F entries. 


. Record in Column H the sum of the Column C entry and 


zi 
Record in Column J the difference between the Column H 
and Column F entries. 


. Record in Column K the quotient obtained by dividing the 


Column G entry by the Column J entry. 


. Record in Column L the square root of Column K. The 


values in Column L are the residual reliabilities of the 
tests to be used in Table 5. 


11. Again apply the shrinkage formula as in Step 5, using Table 


6. 
a. 


b. 


Enter in the test number column the test number of the 


second selected test. 
2 


2 


Enter in Row 2, Column B the quotient z.° 


2° 
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12. 


. Record in Column D the quotient 
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. Record in Row 2, Column C, the difference between the 


Row 1, Column C entry and the Row 2, Column B entry. 
N-1 





-. Record in Column E the product of the Column . and 


Column D entries. 


. Record in Column F the difference between 1 and the 


Column E entry. 


. Record in Column G the square root of the Column F en- 


try. This value for R is the new shrunken multiple cor- 
relation coefficient. If it is smaller than the preceding RF, 
the second test has added more chance error than actual 
validity. In this event, work should be stopped and only 
the first test should be used to predict the criterion. If 
the R is larger than the preceding one, the addition of 
tests to the battery must be continued. 


Continue the Doolittle procedure, using Table 7. 


a. 


Enter in the 2. Row of Table 7 the correlation coefficients 
of the second selected test with every other test, as well 
as with the criterion. Remember to reverse the sign of 
the correlation of the second selected test with the cri- 
terion. 


. Record in the check sum column the algebraic sum of the 


a. entries. 


Draw a vertical line through the b. and ec. Rows for the 
first selected test. 


. Record in the b. Row the sum of each a, entry and the 


product of the b, entry of the same test and the c, entry 
for the second selected test. Formula: b. =a, + [b, (each 
test) X c, (second selected test) ]. The entries in the Cri- 
terion and Check Sum Columns are determined by the 
same method. 


. There are three checks in the b. Row: 


(1) The entry for the second selected test should equal 
the Z, entry for the same test in Table 3. 

(2) The entry in the criterion column should equal the 
V. entry of the second selected test in Table 2 

(3) The entry in the check sum column should equal the 
sum of all the other entries in the b. row. 








13. 


14. 


15. 


16. 


17. 


18. 
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f. Record in the c, Row the product of each b. entry and the 
negative reciprocal of the b. entry for the second selected 
—l 





test. Formula: c. = b. (each test) (second selected 


test). 
g. There are three checks in the c, Rows: 
(1) The c. Row entry of the second selected test should 
be —1. 
(2) The c, Row entry in the check sum column should 
equal the sum of all the other c, entries. 
(3) The product of the b. and c, entries of the criterion 


2 
value in Table 6, Col- 


2 





column should equal the 
2 


umn B, Row 2. (Disregard sign). 


Draw a vertical line under the second selected test in Tables 
2 and 3. 


The V; entries are calculated exactly as in Step 8 except that 
all subscripts should be increased by 1 to apply at this point. 
Formula: V; = V. + [b2(criterion) X c. (each test) J 


The Z; entries are calculated exactly as in Step 9 except that 
all subscripts should be increased by 1 to apply at this point. 
Formula: Z; = Z, + [b.(each test) < c. (same test) ] 


Select as the third test in the battery, the test which will 
give the greatest return in validity per unit of testing time. 
This is accomplished in the same manner as in Steps 10 and 
4, 


Apply the shrinkage formula as in Step 5 using Table 6. If 


the R is larger than the preceding one, a fourth test must 
be considered for inclusion in the battery. 


Continue the Doolittle procedure using Table 7. 

a. Enter in the a, Row of Table 7 the correlation coefficients 
of the third selected test with every other test as well as 
with the criterion, the latter with reversed sign. 

b. Record in the check sum column the algebraic sum of the 
a, entries. 

c. Draw a vertical line through the b, and ce; entries of pre- 
viously selected tests. 

d. The formula for the b,; entries is: b; =a, + [b, (each test) 
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X ¢, (third selected test)] + [b. (each test) X c. (third 
selected test) ]. 
e. The formula for the c, entries is: c; = b; (each test) X 


—] 
ea (third selected test). 


f. The same checks apply here as given in parts e. and g. of 
Step 12. 


19. The-values for V, and Z, are determined as in Steps 14 and 
15 with the required change in subscript. Formulas: V,= V; 
+ [b, (criterion) < c, (each test)]; 7, = Z, + [b;, (each 
test) < c; (same test) ]. 


20. Repeat the necessary calculations in steps 4 through 10 un- 


til an R smaller than its preceding one is obtained, until all 
the tests have been included in the battery, or until the po- 
tential gain from adding further tests seems uneconomical. 


Test Selection by Modified Method 

Content of Demonstration Battery.—The battery of tests used 
here for demonstration purposes includes 15 tests from among a 
group of 46 that had been administered, at various stages of their 
training, to 407 A.A.F. rated bombardiers who were candidates for 
radar observer training. The 15 tests were found to be correlated 
with the criterion at least at the five per cent level of significance. 

The criterion was a composite of course grades which were based 
entirely on standardized test and performance check scores. Although 
no method was available for directly determining the reliability of 
the composite course grades, an approximate reliability was deter- 
mined using the split-half method. The composite score was divided 
into two parts which were made as nearly alike as possible as to con- 
tent. For a sample of 278 men from one training station, the corre- 
lation between the two “halves” of the course grade was .27, which 
became .43 when corrected for double length. 

This approximated value for the reliability of the criterion does 
not enter into the multiple correlation calculations but it does serve 
as an approximate indication of the upper limit that the multiple R 
can be expected to reach.* The size of the obtained multiple R would 
therefore tend to indicate that the battery would do as good a pre- 


diction job as could be expected under the circumstances. 

*Theoretically, the square root of the obtained reliability represents the up- 
per limit of the coefficient of validity for a test. See Lindquist, E. F. A first 
course in statistics. New York: Houghton-Mifflin Co., 1942, p. 224. 
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The fact that eleven of the 15 tests are included in the multiple 
FR before it starts to shrink can be at least partially accounted for by 
several reasons, among them being the relatively large size of the 
sample and generally low intercorrelations between the tests. The 
relatively low correlations of the individual tests with the criterion 
can be explained, to a certain degree at least, by the narrowed range 
of talent in the sample caused by the men being subjected to at least 
three selection procedures for which corrective data were not avail- 
able, and by the relatively low reliability of the criterion. 


Application of Modified Method 


2 


Referring to Table 2, Series 1, Test 8 has the largest = ; 


1 





which is .0324. This test requires ten minutes of testing time. There 
are three tests, Numbers 10, 13, and 14, which require less testing 
time; hence they should be checked to determine whether one of them 
will furnish a higher rate of return in validity per unit of testing 
time than Test 8. Reference to Table 5, Series 1, wherein the calcu- 
lated time-adjusted values of these tests appear, demonstrates that 
Test 8 for 7.5 minutes retains the highest adjusted value (in com- 
parison with Tests 10 and 14) and is thus selected as the first test 
for inclusion in the multiple R. Test 14 was given a time-adjusted 
value for 7.5 minutes because it was anticipated that this test would 
not furnish as great a rate of return in validity per unit of testing 
time as Test 10. However, a check must be made, and 7.5 minutes 
was a convenient point, since the time-adjusted values of Tests 8 and 
10 were calculated at that time. It was not necessary to calculate a 


. 


value at five minutes 





time-adjusted value of Test 13 because its 


1 

testing time is considerably smaller than that of Test 14 at just 5.5 
minutes and therefore would remain smaller when adjusted. As men- 
tioned in Step 13 of the procedure previously presented, after a very 
few such calculations it is quite simple to anticipate in most cases 
which test and testing time will be selected, although of course this 
must be checked mathematically. 

In Series 2, Table 2, Test 10, requiring 7.5 minutes testing time, 

2 


v 
has the largest —— value. Tests 18 and 14, which require 5 and 5.5 
minutes of testing time, respectively, have values that will not match 
the value of Test 10 at 7.5 minutes, as was indicated by the time- 
adjusted values determined in Series 1. Therefore Test 10 is the sec- 


ond test included in the battery. 
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- 
value and has a shorter 





In Series 3, Test 4 has the largest 


3 
testing time than Tests 1 and 5, which are the only ones having values 
appreciably rivaling the value of Test 4, and therefore Test 4 is the 


third test selected for inclusion in the multiple. 
2 


V, 
In Series 4, Tests 1 and 5 have by far the largest Zz. values. 


4 


When a time-adjusted value is calculated for the latter test at 18 min- 
utes for comparison with the former, Test 1 still has the largest time- 
adjusted value. 

In Series 5, Tests 5 and 15 are equated for testing time, which 
shows that Test 15 has the largest time-adjusted value and is there- 
fore the fifth test included in the muliple R. 

In Series 6, Tests 5 and 13 are equated for testing time, with 
Test 5 again having the smaller time-adjusted value, indicating that 
Test 13 should next be included in the battery. 

In Series 7, Test 5 is equated in time value to Test 14. At this 
point it has the largest time-adjusted value and therefore is the sev- 
enth test selected. 


value with the shortest 





In Series 8, Test 15 has the largest 


testing time, so it is included next in the multiple R without question. 

In Series 9, Test 7, a 35-minute test, is equated with Test 2 at 
20 minutes but retains the highest time-adjusted value and is thus 
the ninth test selected for inclusion in the battery. 

In Series 10, Test 2 is compared with Test 6 and again has the 
smaller time-adjusted value, indicating that Test 6 should next be 
selected. 

In Series 11 and 12 there is no question concerning which test 
should be added to the multiple. 


Comparison of Results from the Two Methods 

The results of the application of the regular Wherry-Doolittle 
test selection method to the demonstration battery are presented in 
Table 8. A comparison of the results obtained from the applications 
of the two methods is presented in Figure 1. In this figure cumula- 
tive testing time is plotted on the base line and the multiple R on the 
vertical axis. The first divergence of results from the two methods 
occurs when the fourth test is added to the battery. 

Using the Wherry-Doolittle method, when Test 5 is added the 
multiple correlation becomes .283, requiring 63.5 minutes of testing 
time. With the modified method, Test 1 is added as the fourth test, 
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Comparison of Results from the Application of Original and Modified Wher- 
ry-Doolittle Method to a Demonstration Battery 


making the multiple R exactly the same, but only 45.5 minutes of 
testing time is required. This difference of 18 minutes of testing time 
would be of considerable importance in an industrial testing situa- 
tion. Adding a fifth test would produce a multiple R of .299, with 
just 60.5 minutes of elapsed testing time using the modified method. 
Again with the modified method, a multiple R of .309 is obtained 
using just two minutes more of testing time than is necessary to ob- 
tain an R of .283 using the Wherry-Doolittle method. This advan- 
tage of greater return in validity for elapsed testing time is main- 
tained by using the modified method, as can be seen by reference to 
Figure 1, until the eighth test is added to make the obtained R .3395 
by the Wherry-Doolittle method and .3393 by the modified method. 
The same three tests are subsequently added when using both proce- 
dures to bring the R to .3484 before it begins to shrink when another 


test is added. Since the R increases only from .339 to .348 with 80 
minutes of additional testing time, it would probably not be practi- 
cal to use the additional tests. 

As a general practice in using a battery of tests selected on the 
basis of a multiple correlation, the number of tests that should be 
used for predictive purposes would depend considerably upon the sig- 
nificance of the difference between succeeding R’s obtained as tests 
were added to the battery. Since we are interested here only in dem- 
onstrating the possibility of identifying test batteries which will re- 








152 PSYCHOMETRIKA 


quire a minimum amount of testing time, it was not deemed neces- 
sary to present such differences for the battery used as an example. 


Conclusions 


The modified Wherry-Doolittle test selection method as presented 
will produce for a given battery of tests a multiple correlation with 
the maximum possible value requiring the minimum possible testing 
time. By using this method, it is also possible to determine the larg- 
est multiple correlation that can be obtained from a given battery of 
tests in a particular period of testing time. Such information is valu- 
able for administrative reasons and has considerable economic im- 
portance. 
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TABLE 2 
Worksheet for Selection of Tests for Inclusion in Multiple R, Part 
1—Modified Wherry-Doolittle Method 
- Test “% 
Number 1 2 3 4 5 6 vf 
a Testing a 
Time, 
Minutes 18 20 24 10 36 25 35 
V2 a 
Z, 
V, —.18 —.11 —.12 —.11 —.10 —.10 —.14 
V,? 
Z, .017373 015575 .014528 
V, —.1318 —.0506 —.0534 —.1244 —.1198 —.0568 —.0806 
V;2 
Z; 014651 020762  .014026 
V, —.1206 —.0365 —.0392 —.1423  —.1177 —.0494 —.0733 
V2 
Z, 016494 .016510 0050 
V, —.1278 —.0568 —.0886 —.1274 —0251 —.0069 
Z } 
af .012563 
Z; 
V; —.0623 —.0341 —.1101 —.0050 —.0500 
V2 
Z, 006452 013487 
V. —.0739 —.0368 —.1140 —.0040 —.0573 
V; 2 
Z, .014248 
V, —.0574 —.0274 —.1171 .0028 —.0651 
V," ‘ 
Z, 
Vz —.0521 —.0265 .0197 —.0450 
V,? 
Z, .003467 005504 
y. —.0530 —.0154 .0235 —.0657 
Vio" 
Ze 002620 008990 
Vio —.0458  —.0056 .0529 
Wage 
Ss .003154 
F. —0501  —.0104 
V2? 
Zs 
wie —.0029 
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TABLE 2 (Continued) 
Worksheet for Selection of Tests for Inclusion in Multiple R, Part 

1—Modified Wherry-Doolittle Method 

__f Test — 
Number . 9 10 11 12 18 14 15 
1§ Testing 7 

=" 6©Time, 

Minutes 10 15 1.5 30 15 5 5.5 15 

ee 

i 0824 0289 0081 0169 

y, =i. oath in — a | | —.09 wing er 
~.14 Vs 

Z, 017462 004283 .006091 .011965 

V, —.0606 —.1286 —.0784 —.0886 —0648 —.0742 —.1076 
0806 | Vs" 
a —.0397 —.0672 —.0619 —.0588 —.0486 —.0996 
V; 
0733 fF Vs? 

Z, 002438 004232 .008285  .006184  .018212 
an V, —.0459 —0612 —0600 —O71S —0O718 —12104 
-006: V;? 

. _ .007597 011445 

5 

V, —.0320 —.0398 —.0888 —.0848 —.0745 —.1044 
0500 F Ve" 
Z, 007965 .006415 
V. —.0221 —.0369 —.0301 —.0868 —.0734 
0573 | V:" 

a, .005576 

a ws —.0095 —.0440 —.0199 —.0683 
0651 | ’s_ 

oe .009648 

V, —.0004 —.0062 —.0196 —.0883 
0450 F Vs? 

Z, 
055mf CV 0101 —.0080 —.0063 
0657 ) V0? 

Z,, “ 

: Vio 0083 0218 .0048 

Z,, 

Vi, .0102 0056 .0031 

Zs 000143 

,.. .0309 0018 = .0119 
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158 PSYCHOMETRIKA 
TABLE 6 
Worksheet for Application of Wherry Shrinkage Formula— 
Modified Wherry-Doolittle Method 
A B C D E F 
Testing Cumula- Test V,2 Nes 
Time, tive Nun -'/ Sy a ; 
Minutes Time, _ ber Z, 2 N-— Kk? Rk 
Minutes (1—B) (CXD) (1—E) 
0 1 
10 10 8 1 .0324 .9676 1.00 .9676 .0324 
7.5 17.5 10 2 01746 .95014 1.002469 .952486 .047514 
10 27.5 4 38 .02076 .929388 1.004950 .938980 .066020 
18 45.5 1 4 .01649 .91289 1.007444 .919686 .080314 
15 60.5 15 5 .01144 .90145 1.009950 .910419 .089581 
5 65.5 13 6 .00796 .89349 1.012469 .904631 .095369 
36 101.5 5 7 .01425 .87924 1.015000 .892429 .107571 
5.5 107.0 14 8 .00964 .86960 1.017544 .884856 .115144 
35 142.0 7 9 .00551 .86409 1.020101 .881459 .118541 
25 167.0 6 10 899 .86010 1.022670 .879598 .120402 
20 187.0 2 11 .00815 .85695 1.025253 .878591 .121409 
15 9 12 .00014 .85681 1.027848 .880670 .119330 


1800 
.2180 
.2569 
2834 
2993 
3088 
0280 
3393 
3443 
.3470 
3484 


3454 








> 
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TABLE 7 
Worksheet for the Doolittle Solution of Normal Equations 
Wherry-Doolittle Method 











Test 

Number 1 2 3 4 5 6 | 8 9 

a, —_— _— —_— —_— _— — — _-_ — 

8 b, -.01 353 ST -.08 -11 24 838 1.00 .33 

¢, 01 -—33 —.37 .08 11 —.24 -33 -1.00 -.33 

a, .08 18 19 —.15 -.01 it 18 20° 20 
10 b, 0823 .1041 .1049 -—1316 .0153 .0548 .0541 | 1541 
Cy -.0869 -.1099 -—.1108  .1889 -.0162 -.0579 -.0571 | —.1627 

a, ~—.06 -—18 -.04 1.00 —.06 14 01 -.08 -.09 
4 b, -.0494 -.1391 .0042 .9753 -.0667 .1668  .0439 |  -0422 
Cy 0507 .1426 -.0443 -1.00 .0684 -.1710 -.0450 | .0433 

a, 1.00 —.03 .04 —.06 14 15 14 -01 .12 
1 b, .9902 -.0428 .0348 | 13842 .1561 .1308 | .1078 
c, -1.00 .04382 -.0351 -.1855 -.1576 -.1321 | -1089 

a, .07 —.03 .05 -.11 —.04, .05 OGG 218%. -2F 
15 b. | —.1056 -.0249 -.0356 .0086 -.0669 .0905 
Cc, | 1109 .0261 .03874 -.0090  .0703 ~.0950 

a, .04 .26 16 —.22 —.03 .08 —.04 a. oe 
13 b, | 1789 .1024 ~.08385  .0741 -.0847 | .1360 
Ci -.1913-— .1095 .0358 -.0792 .0906 —.1455 

a, 14 01 -.08 —.06 1.00 12 15 -11 .05 
5 b, | .0484 .0073 .9624 .1887 .1652 | .0746 
Cc, | —.0451 -—.0076 | —1.00 -.1441 -.1717 | -.0775 

a, .0001 = .14 .24 -.21 -.19 07 -.12 Si: 23 
14 b, -.0086 .1013 | | .0345 -.1892 | .0963 
C, .0106 —.1253 | -.0427 .2340 | —1191 

Gy 13 19 we 01 15 Ri | 1.00 8 ~—L08 
7 b, | .0864 .1171 | | 8511 .7887 | 0216 
Cy | —.1102 —.1494 | | —.4480 -1.00 | .0277 

A, 15 AT 28 14 BS 4 1.00 AT at - 8 
6 Dis, | .0575 .0632 | | -7014 | |  -.0247 
” | ~.0820 -.0901 | | —1.00 | | .0352 

a,, 03 1.00 .29 -.18 01 AZ 19 S38 47 
2 b,, | .7958 .1188 | | | | | 3289 

| 


-1.00 -.1498 | | | -.4188 


biti saith 
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TABLE 7 (Continued) 
Worksheet for the Doolittle Solution of Normal Equations 


Wherry-Doolittle Method 

















Test Check 
Number 10 11 12 18 14 15 —C Sum 
a, — oe — = = <= = == 
8 b, we 12 48 a4 ol 18 -.18 3.68 
C, —.23 —.12 —.48 —.14 -—.o1 -.18 18 -3.68 
a, 1.00 al Zi .08 .26 10 -.17 2.64 
10 b, 9471 0824 .1596 .0478 -1887 .0586 -.1286 1.7936 
c, 1.00 -.0870 -.1685 -.0505 -.1992 -.0619  .1358 —1.8938 
a, —.15 .02 -.02 —.22 —.21 -.11 -.11 —.1600 
4 b, | 0410 -.0406 -.2022 -.1590 -.0875 -.1423 .8835 
C. | -.0420 -.0416 .2073 .1630 .0897 .1459 -.3932 
a, 08 AT 14 03 0001 .07 -.18 1.8501 
1 b, | -1661 .1830 .0270 -.0218 .0623 -.1278 1.7505 
C, | -.1677 -.1848 -.0273  .0215 -.0629 .1291 -—1.7678 
a, 10 .06 18 .08 .09 1.00 —.14 1.6601 
15 b. | .0265 0790 -.0180 .0096 .9522 —.1044 .8110 
£, -.0278 -.0830 .0189 -.0101 -1.00 1096 -.8517 
a, .08 —.06 18 1.00 14 .08 -.09 1.8700 
13 b, | -~0765 .1110 .9850 .0549 | -.0863 1.8113 
C. .0818 -.1187 -1.00 —.0587 | .0923 —1.4024 
a, -—.01 .o2 —.04 -.08 -.19 —.04 -.10 1.1800 
5 b, | 3104 .0019 | ~.1646 | ~1171 1.4221 
c, —.38225 -.0020 .1710 | 1217 1.4777 
a, 26 -.08 30 14 1.00 .09 ~.13 2.1001 
14 -B, l -0161 1219 | .8080 -.0884  .8602 
Cy .0199 -.1508 | —1.00 | .1093 —1.0640 
a, 18 48 27 -.04 12 0001 -.14 8.1601 
7 b, | 3498  .1269 | | | -.0657 1.7277 
re -.4463 -.1619 | | | .0888 —2.2045 
Ao 11 AT 24 .08 07 .05 —.10 8.5800 
6 Ris .2088 §©.0158 | | | 05380 1.0747 
i. —.2970  .0225 | | | -.0756 1.5322 
a, 18 .02 33 26 14 —.03 -.11 3.04 
4 b., | -.0679  .13890 | | -.0501 1.2643 
Cc 0858 -.1747 | | .0630 —1.5887 
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TABLE 8 
Worksheet for Application of Wherry Shrinkage Formula— 
Wherry-Doolittle Method 

















A B Cc D E F G 
Testing Cumula- Test V2 N==1 

Time, tive Num-M — = ” ne 

Minutes Time, ber Z, Kk? N—M K? R? R 
Minutes (1—B) (CXD) (1—E) (VF) 

0 1 

10 10 8 1 .0824 .9676 1.00 -9676 .0324 18 
7.5 17.5 10 2 .01746 .95014 1.002469 .952486 .047514 .2180 
10 27.5 4 3 .02076 .92988 1.004950 .923980 .066020 .2569 
36 63.5 5 4 .01651 .91287 1.007444 .919665 .080335 .2834 
15 78.5 15 5 .01406 .89881 1.009950 .907753 .092247 .3037 
18 96.5 1 6 .01084 .88797 1.012469 .899042 .100958 .38177 
5.5 102 14 7 .01069 .87728 1.015000 .890439 .109561 .3310 
5 107 13 8 .00771 .86957 1.017544 .884768 .115232 .3395 
35 142 if 9 .00552 .86405 1.020101 .881418 .118582 .3444 
25 167 10 .00399 .86010 1.022670 .879598 .120402 .3470 
11 .00815 .85695 1.025253 .878591 .121409 .3484 





6 
20 187 2 
9 00014 .85681 1.027848 .880670 .119330 .3454 


ay 
bo 
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DON LEWIS. Quantitative Methods in Psychology. Ann Arbor: Edwards 
Brothers, 1948. Pp. v + 286. 


The purpose of this volume is best characterized in the words of the author: 
“This book was prepared specifically as a text for an advanced graduate-level 
course in psychology, in the experimental area .... The contents of the present 
course, called Quantitative Methods in Psychology, are in general the same as the 
contents of this book. The course does not replace the usual course in statistics. It 
is given in addition to six other courses (three of them quite advanced) in the 
general area we call ‘quantitative methods and statistics.’ ” 

Since there are only ten chapters, the chapter headings are here listed to give 
the reader a quick view of the organization of the book. 


Chapter 1—Variables, Constants, and Functional Relationships 

Chapter 2—Fitting Curves to Empirical Data: 
I. Linear Functions 

Chapter 83—Logarithms 

Chapter 4—Fitting Curves to Empirical Data: 
II. Complex Functions 

Chapter 5—Differentiation 

Chapter 6—Integration 

Chapter 7—The Normal Curve 

Chapter 8—Distribution Functions 

Chapter 9—Applications of Equations 

Chapter 10—Goodness of Fit 


The techniques discussed in the various chapters are thoroughly illustrated 
with concrete psychological data. Exercises appended to the chapters are well- 
chosen and great enough in number to give the student ample practice. The 
mathematical chapters should serve as adequate review for the student who has 
previously been through the calculus. We doubt that they would be adequate for 
the more typical psychology student who has not gone beyond college algebra, 
though this could hardly be considered the fault of their author. 

Perhaps the most satisfactory chapters from our point of view were chapters 8 
and 9. In the former the author describes the binomial and Poisson distributions, 
then: discusses the relationships between the normal distribution and the distri- 
butions of t, chi square, and F in possibly the most meaningful manner available 
in psychological statistics. In the latter chapter the applications of equations 
both to physical science data and to psychological data are discussed. The seven 
pages devoted to Galileo and Newton, in this chapter, are quite illuminating. An 
expansion of this material would be much more useful to our students than courses 
in the philosophical history of psychology. 3 

Since this is a preliminary edition, we feel free to suggest additions as well as 
corrections at this time. The reviewers, for instance, would have found a section 
on units of measurement in psychology very useful. The problem of units is 
certainly critical in the determination of functional relationships between vari- 
ables. There has also been some degree of neglect of units of measurement in the 
preparation and discussion of the figures. There seem to be two rather different 
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approaches to the matter of graphing. One is aesthetic. The other is to attempt 
to portray essential relationships. In his frequent neglect of origins, and in his 
selection of units for the graphs, the author comes closer to the first of these two 
approaches. 

It would help the reader, we believe, if the psychological implications of the 
various equations were sometimes discussed more fully. A learning curve equa- 
tion on page 16, for example, has a negative value for the additive constant (y- 
intercept). It is surely of interest that the equation implies that learning starts 
below zero. 

We also suggest that definitions could be emphasized to advantage by slight 
changes in typography and format. As it is, important definitions are frequently 
hidden in the text. We also found a few important terms introduced without 
definition; e.g., antilogs, on page 36. 

There is a little confusion with regard to an application of chi square. On page 
258 a paragraph is devoted to a discussion of a fundamental error made by certain 
other experimenters in the application of chi square to a series of 132 scores made 
by 33 subjects. Independence of separate scores had been implicitly assumed, 
but not tested. Previously, on page 162, the author made a similar mistake, in- 
volving 1600 scores from 320 animals, after having made a test which suggested, 
at the 5% level of confidence, that successive turns on the maze were not inde- 
pendent of each other. A further illustration on page 164 also involves multiple 
responses from individual animals, but a previous chi square test had suggested 
that the trials were statistically independent. 

Other points, briefly noted, are as follows: (1) Some of the illustrations of 
constants (p. 1) are not particularly apt. The constancy of an individual’s I.Q. 
is really quite a different sort of constancy than is the value of pi. (2) Defini- 
tions of dependent and independent variables (p.2) do not cover the many situa- 
tions in which their designations are arbitrary. (83) Linear correlation is illus- 
trated only as a measure of goodness of fit for scatters of means. The properties 
of r when used in this way are not distinguished from the properties of 7 when 
it is based upon scatters of individuals. For example, 7 in the first case is a 
function of the number of observations, whereas r in the second case is independent 
of N. (4) In the discussion of the normal curve as an outgrowth of the law of 
error (p. 145), the author does not make explicit all necessary assumptions re- 
garding the nature of the errors. (5) A negative exponent (p. 186) should be 
changed to a positive when the accompanying expression is moved from the nu- 
merator to the denominator of the fraction. (6) The application of the F-test on 
page 193 involves doubling the p-value obtained from the table. (7) In the expres- 
sion for the standard error of a difference between correlated means on page 194, 
the cross-product term is omitted. It is brought back implicitly in the next step by 
shifting to the distribution of differences. (8) The t-test suggested for use with in- 
dependent distributions having unequal variances (p. 196) has certain merits, but 
it involves chance correlations between values paired at random. With samples 
of the size illustrated, chance correlations could markedly influence the final re- 
sult. (9) The experimental advantages and disadvantages of the various t-theo- 
rems are largely neglected (pp. 189-196). (10) There are two significant omis- 
sions from the discussion of chi square. One is the failure to mention Yates’ 
correction for continuity. The second is the failure to point out the suspicion with 
which one should view a very high chi square p-value, when testing goodness of 
fit. A chi square p greater than .95 will occur by chance no more often than a p 
less than .05, and should lead one to a careful check of computations and as- 
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sumptions. (11) A few typographical errors are evident, but are no more fre- 
quent than are expected in a lithoprinted publication of this nature. 

The author has brought together a great deal of material not available else- 
where. Dr. Lewis has a much more comprehensive treatment than Guilford at- 
tempted in a somewhat comparable section of Psychometric Methods. The re- 
viewers believe that this volume constitutes a major contribution to experimental- 
theoretical psychology. They further believe that advanced graduate students 
with these interests should be offered a course of the type described by this text. 


Stanford University Lloyd G. Humphreys 
. Lyle V. Jones 


EGON BRUNSWIK. Systematic and representative design of psychological ex- 
periments. Berkeley and Los Angeles: University of California Press, 1947. 
Pp. 60. 


Experimental psychologists have long striven to emulate the rigorous control 
and isolation of variables in the manner of the physical sciences. Brunswik’s mono- 
graph presents a thoughtful and challenging critique of psychological experimen- 
tation conceived in the image of physics and a call for a new type of experimental 
design. Although most of the illustrations are drawn from the area of percep- 
tion, the thesis of the monograph has wide implications for the total field of 
quantitative research in psychology. 

Representativeness is the concept which provides the key both to Brunswik’s 
criticism of traditional experimental design and to his plea for a new method- 
ology. The classical type of experiment usually employs only a few carefully 
chosen values of the stimulus variable: a set of sizes, weights, or brightnesses. 
The range of values of the variable which the organism encounters and re- 
sponds to in its natural habitat is vastly larger and more diversified. From what 
Brunswik calls an ecological point of view, such a choice of variables fails to 
yield a representative sample of behavior. As a consequence, the results of many 
systematic classical experiments lack generality and cannot easily be translated 
into general laws about the organism’s response to the variable. 

Lack of representativeness, Brunswik further argues, also results from the 
manner in which the covariation of variables is handled in the classical experi- 
ment. Even though the observed behavior may be a joint function of several de- 
terminants, these determinants are not allowed to vary together in their natural 
(i.e., ecologically effective) manner. Instead, variables are “artificially tied,” 
“artificially interlocked,” or “artificially untied.” The artificial tying of variables 
is illustrated by the conventional Galton-bar experiment, or for that matter, a 
large number of classical psychophysical situations. In such a situation, the 
physical size of the stimuli and their retinal projections are not allowed to vary 
independently; due to the artificial perfect correlation, it is impossible to assess 
the relative contribution of these two factors to perceived size. Artificial inter- 
locking of variables is exemplified by the conventional size-constancy experiment. 
With the physical size of the stimulus constant, the magnitude of the retinal pro- 
jection is allowed to vary over a few fixed values. Finally, variables are artifi- 
cially untied when a large number of potential determinants is successfully held 
constant and only one factor is allowed to vary. When personality traits are 
judged, for example, subjects may be required to wear identical clothing and to 
assume the same position. The probably joint dependence of the judgment on 
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facial expression, posture, and clothing habits is arbitrarily destroyed. To the 
extent that variables cannot be held constant, they are frequently assigned to 
residual variance. 

In Brunswik’s analysis, the procedures of mental testing represent, as it 
were, the reverse side of the classical coin. Here individual differences are the 
focus of interest and quantitative treatment. But the stimulus (test) is held as 
constant as possible, and lack of stimulus variation precludes ecological generality 
of results. Neither the systematic, quasi-physical type of experimental design nor 
the differential statistics of mental testing thus meet Brunswik’s requirement of 
representativeness. 

These criticisms do not, as the author fully recognizes, apply with equal force 
to the total range of experimental designs used in psychological research today. 
Brunswik emphasizes the gradual but steady development of perceptual research 
away from single-variate investigations to the study of field dynamics or “multi- 
dimensional psychophysics.” Multivariate experiments, however, are only a first 
step toward truly representative design. To achieve representativeness, it is nec- 
essary to apply the full freedom (and restrictions) of sampling to stimulus situa- 
tions and objects. Just as a well-selected group of subjects must be representative 
of the population from which it was drawn, so a sample of stimulus situations 
must be representative of the total range of stimulus values in the environment 
of the organism. Experiments on size constancy again provide the paradigm. 
Instead of confining size judgments to a limited set of test stimuli at a few ar- 
bitrarily selected distances, Brunswik and his students have obtained repre- 
sentative samples of size judgments while their subjects went about the routine 
of their daily lives. A high degree of constancy in the appearance of physical 
bodies was shown to hold over a wide range of stimulus situations. At the same 
time, the importance of the attitude under which the judgments are made 
(whether a “betting” or a “critical” attitude) was demonstrated. This repre- 
sentative field study not only confirmed the results of laboratory experimentation 
but also served to give them a high degree of situational generality. 

Situations as well as subjects, Brunswik affirms, must be subjected to repre- 
sentative sampling. If such a program were to be carried out, the experimenter’s 
tools of statistical analysis must be adapted to the changing theory of design. 
The author suggests that correlational analysis may go a long way toward meet- 
ing this need. 

The coefficient of correlation thus far has been largely a tool of differential 
psychology. A sample of subjects is exposed to two test situations or measure- 
ments and the covariation of these measurements is ascertained. The crucial 
point for Brunswik is that it is subjects who are sampled and that the range of 
situations over which the measurements are taken is highly restricted. As far as 
the logic of correlation is concerned, it is possible to sample situations and to cor- 
relate (1) different responses to the same situation,,(2) responses and stimulus 
characteristics, and (3) different stimulus characteristics. To exemplify each of 
these, (1) we may correlate ratings of intelligence and ratings of personal ap- 
pearance given by a single judge to a sample of individuals, (2) physical size 
(or retinal size) may be correlated with perceived size for a sample of objects, 
and (3) physical size may be correlated with retinal size. The important point is 
that such uses of the correlation coefficient are object-centered rather than sub- 
ject-centered. As Brunswik himself puts it, “individuals and test situations have 
shifted places” (p. 34). Brunswik suggests that this use of the correlation co- 
efficient may lead to a type of “intra-personal” factor analysis. In proposing that 
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individuals and test situations shift places, however, Brunswik does not mean the 
kind of shift involved in going from conventional factor analysis to inverse factor 
analysis. In conventional factor analysis, the correlation between two test vari- 
ables or “features” is obtained over a sample of individuals with the situation held 
constant. What Brunswik suggests is that we obtain correlations between “fea- 
tures” for a single individual over a sample of situations. This type of analysis 
would, then, be an analysis of “intra-individual” correlations. Thus it may become 
possible to isolate the basic situational dimensions of perception by factorial 
methods. Any subject could be characterized in terms of situational factors which 
describe the dimensions of his perceptual world. 

Object-centered correlational analysis leads Brunswik to a reexamination of 
the concepts of reliability and validity. He lays particular stress on indices of 
reliability which are based on ccrrelations between responses: intra-individual 
and inter-individual reliability. Intra-individual reliability refers to the con- 
sistency of responses given by the same observer to the same sample of situations. 
It measures the stability of an individual’s response to his environment. Inter- 
individual reliability—the agreement of observers in the presence of a given stimu- 
lus situation—measures the “ecological reliability of a response,” i. e., the extent 
to which a stimulus has comparable psychological consequences for different or- 
ganisms. Such measures of reliability again are object- rather than subject-cen- 
tered. The concern is not with variability in a large sample of subjects but ra- 
ther with the consistency of responses (given by one or two subjects) to an object 
or representative sample of objects. 

Brunswik’s conception of validity is rooted in his general theory of percep- 
tion. Validity of a response refers primarily to what the response achieves for the 
organism. We must recall that Brunswik has analysed perceptual responses in 
terms of the “intentions” of the organism in responding to an object in the en- 
vironment. Thus, in judging size, the observer may intend to attain the actual 
physical size of the object. The correlation between physical size and perceived 
size then gauges the extent to which this intention is realized. This correlation 
measures the “functional validity” of the judgment. Under a different attitude, 
the observer may intend to attain not the physical size but the retinal size of the 
object. In this latter case, the correlation between retinal size and judged size 
provides the measure of functional validity. This treatment of validity clearly 
identifies Brunswik as a modern functionalist. His fundamental concern as a be- 
havior theorist is with the mechanisms mediating successful adjustment to the 
environment. , 

Perceptual experiment and theory provide Brunswik with the bulk of his il- 
lustrative material. He hopes, however, that the methodology for which he argues 
will be extended to other fields of inquiry, particularly learning. Learning ex- 
periments, he believes, no less than perceptual experiments, must meet the chal- 
lenge of representativeness. 

Brunswik aptly describes his approach to experimental design as ““probabilis- 
tic functionalism.” The watchword is representative sampling of objects and situ- 
ations. “Psychology is conceived of as a fundamentally statistical discipline 
throughout its entire domain, with ‘functional validity’ taking its place alongside 
traditional test validity” (p.56). The generality and applicability of the findings 
will depend primarily on the representativeness of the samples. 

Such, in broad outline, is Brunswik’s thesis. The cogency of his argument 
hinges on the assumpiion that representativeness (ecological validity) is the 
touchstone of experimental design. Often systematic laboratory experiments fail, 
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indeed, to meet this criterion. Many such experiments are aimed at the construc- 
tion of models and the isolation of basic mechanisms of behavior with the aid of 
these models. Representativeness then becomes largely a problem of application, 
and essentially parametric study. Underlying this philosophy of experimentation 
is the belief that simplification for the sake of analysis—the artificial restriction, 
isolation, and interlocking of variables—does not preclude the discovery of highly 
general, ecologically valid cause-effect relationships. Brunswik’s own representa- 
tive study of size constancy, for example, bore out in large measure the results of 
artificially restricted laboratory experiments. 

Representative sampling of situations can be effective only if the universe of 
situations is sampled with respect to some crucial characteristic. It is in the dis- 
covery of such crucial characteristics that the systematic, classical type of ex- 
periment may well be propadeutic to Brunswik’s type of representative design. 
It seems that representative surveys of behavioral situations cannot replace sys- 
tematically controlled experiments; the two approaches must proceed side by side. 
Nor is the representative sampling of objects and situations necessarily and 
always superior to the representative sampling of subjects. It may often be impor- 
tant te vary the nature of the responding organism over a wide range, to sample 
different life histories and past experiences. In many respects, organisms are not 
interchangeable, and the sampling of situations with the nature of the organism 
constant or quasi-constant may fall short of representativeness. 

We shall do well indeed to heed Brunswik’s plea for representative sampling 
of behavioral situations. At the same time it must be hoped that the extension of 
statistical surveys of behavior would not weaken our concern with theoretical 


(albeit artificial) models. 
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